Marown/yacy_search_server
1
0
Fork 0
mirror of https://github.com/yacy/yacy_search_server.git synced 2024-09-21 00:00:13 +02:00
Code Issues Actions Packages Projects Releases Wiki Activity
9,756 Commits 1 Branch 20 Tags 836 MiB
92d3f71b16
Commit Graph

9756 Commits

This Branch
This Branch
All Branches
Author SHA1 Message Date
reger
9ef1fd9bac fix: enable use of solrcore.properties for property substitution of solrconfig.xml 2013-06-01 05:50:03 +02:00
reger
8a7fcb391d enable use of solrcore.properties for property substitution of solrconfig.xml 
- move setting of system property solr.directoryFactory=solr.MMapDirectoryFactory to solrcore.properties
- add check of os.arch for 64bit system, if it fails use default/solrcore.x86.properties (if exists) as solrcore.properties
 
reason: on 32bit MMapDirectoryFactory may fail with.....
Caused by: java.io.IOException: Map failed
	at sun.nio.ch.FileChannelImpl.map(FileChannelImpl.java:849)
	at org.apache.lucene.store.MMapDirectory.map(MMapDirectory.java:283)
 2013-06-01 05:43:08 +02:00
Michael Peter Christen
f7e887bf49 added missing class 2013-05-30 16:39:48 +02:00
Michael Peter Christen
eb9d0ba5b1 ranking and boost function update, small bugfixes, better default search 
field for solr
 2013-05-30 16:30:35 +02:00
Michael Peter Christen
5f92c68f1f removed block rank ranking and all YBR files in /ranking 2013-05-30 13:01:22 +02:00
Michael Peter Christen
164603b946 cleanup 2013-05-30 12:47:22 +02:00
Michael Peter Christen
ba793a32c0 added timeout for remote searches of 10 seconds 2013-05-30 12:39:28 +02:00
Michael Peter Christen
1c4c1c0345 try to commit in case of failure which hopefully frees up some RAM 2013-05-30 12:38:54 +02:00
Michael Peter Christen
409d6edf53 Store node/solr search threads to be able to send them an interrupt 
signal in case that a cleanup process wants to remove the search
process. Added also a new cleanup process which can reduce the number of
stored searches to a specific number which can be higher or lower
according to the remaining RAM. The cleanup process is called every time
a search ist started.
 2013-05-30 12:38:15 +02:00
Michael Peter Christen
2a8b99ea82 remove text_t in search result after snippet has been computed to save 
space in search result cache
 2013-05-30 12:35:47 +02:00
Michael Peter Christen
a1644ca0fd new workflow processor in Segment to enqueue indexing documents to solr 2013-05-30 12:34:53 +02:00
Michael Peter Christen
a8dc4346e8 default configuration of MMapDirectoryFactory for solr, increased lock 
timeout, less documents from remote searches (too many results had
easily blocked a peer)
 2013-05-30 12:31:28 +02:00
Michael Peter Christen
0c1a018bbd removed 'later' tactic because it used too much RAM, reduced number of 
soft commits, reduced caching size of search events, ensured that solr
results are processed before connection is closed to keep that stuff not
too long in RAM
 2013-05-29 18:27:27 +02:00
Michael Peter Christen
5344a1c5f7 getting the trash out 2013-05-29 16:09:05 +02:00
Michael Peter Christen
709e9b8ce7 Merge branch 'master' of ssh://git@gitorious.org/yacy/rc1.git 2013-05-29 13:49:42 +02:00
Michael Peter Christen
9e07447d47 added new link for SMW 2013-05-29 13:45:22 +02:00
Michael Peter Christen
3c04dd11de removed dead link 2013-05-29 13:42:38 +02:00
Michael Peter Christen
1eb9626cca less logging 2013-05-29 13:30:32 +02:00
Michael Peter Christen
536fd1450e added new keys for update locations 2013-05-29 13:10:32 +02:00
Michael Peter Christen
281959a2d7 added option to re-boot the embedded solr during run-time. Added also 
API recording for this method so it can be repeated automatically. The
index dump generation is now also available for API recording. Added
some synchronization in backend which was necessary for this.
 2013-05-29 13:09:34 +02:00
Michael Peter Christen
80a7989e8c fixed ClassCastException: [Ljava.lang.Object; cannot be cast to 
[Ljava.util.List; in robots.txt servlet
 2013-05-29 12:02:19 +02:00
orbiter
da621e827e prevent NPE in case RWI is disabled 2013-05-28 16:26:38 +02:00
Michael Peter Christen
c2bcfd8afb Merge branch 'master' of ssh://git@gitorious.org/yacy/rc1.git 2013-05-28 11:39:10 +02:00
Michael Peter Christen
67757b425a use a retry handler with retryCount=0 because we usually expect requests 
to fail if we access non-permanently available resources (peers, web
pages) and want to fail fast without repeating the same request which is
doomed to fail. The previous appearance of http client connection had a
1-2-4-8-second timeout scheme, which caused that connection attempts
lasted for 16 seconds.
 2013-05-28 11:38:45 +02:00
Michael Peter Christen
7300d81f40 include API Table deletion requests to the API recorder 2013-05-28 11:35:56 +02:00
Michael Peter Christen
c2b1075dcf activating pollImmediately in case that DHT receive is off. This will 
cause a much faster search result when running in public robinson mode.
 2013-05-28 10:36:49 +02:00
Michael Peter Christen
d2ade87b49 fixed missing thisaddress in yacysearch.html which caused that the 
opensearch link was not working
 2013-05-28 10:33:41 +02:00
Michael Peter Christen
179d032181 added a (badly formatted) delete button for process scheduler entries 2013-05-27 16:15:58 +02:00
orbiter
888a985dc6 set a higher limit for table copy usage 2013-05-27 15:23:12 +02:00
Michael Peter Christen
2b563debbf javadoc of new multiple-exist test 2013-05-27 13:45:09 +02:00
reger
c03f75ebc3 fix DHT url receive see http://bugs.yacy.net/view.php?id=242 2013-05-26 03:24:32 +02:00
Marc Nause
8fb1b1e290 *) simplified banner creation code 2013-05-25 12:56:43 +02:00
Marc Nause
cd0b5f31b4 *) updated links to description of regex 2013-05-25 11:08:06 +02:00
Michael Peter Christen
8f2d3ce2f9 reduced locking situation in crawler: shifted synchronized location and 
reduced time-out of robots.txt load limit
 2013-05-20 22:05:28 +02:00
Michael Peter Christen
f93501e6e0 nice crawl name if crawl is started with file:// (was: null) 2013-05-20 11:25:26 +02:00
Michael Peter Christen
b4f0cac102 added the reindexing job servlet to the submenu structure 2013-05-20 11:02:21 +02:00
reger
97ab5b90e8 - odt & ooxml (office document) parser correction to add content to fulltext index 
- adjust Junit yacyVersionTest & ParserTest 
- update yacyVersion.combined2prettyVersion to the default 4-digit minor ver.
 2013-05-20 01:50:09 +02:00
Michael Peter Christen
b68fbe7d21 Merge branch 'master' of ssh://git@gitorious.org/yacy/rc1.git 
Conflicts:
	source/net/yacy/migration.java
 2013-05-17 14:13:07 +02:00
Michael Peter Christen
06d3063dc9 - no downcase when using collection modifier 
- removed warnings
 2013-05-17 14:11:10 +02:00
Michael Peter Christen
8dbc80da70 redesign of index.exist-test: this shall now not be done using a single 
id to be tested, but with a collection of ids. This will cause only a
single call to solr instead of many. The result is a much better
performace when testing the existence of many urls. The effect should
cause very much less IO during index transmission, both on sender and
receiver side.
 2013-05-17 13:59:37 +02:00
reger
7f63d3747d more generic field selection for reindex option of documents with disabled fields 
using Luke request to compare config with actual fields in index
 2013-05-15 23:16:32 +02:00
Michael Peter Christen
c91c67c3cd reject bad solr requests 2013-05-15 22:42:05 +02:00
Michael Peter Christen
44e363f37f refactoring of WorkflowProcessor, added process counter, update of 
process counter if an blocking thread dies. Added also a new column in
PerformanceConcurrency_p servlet to show the actual number of concurrent
processes.
 2013-05-13 13:28:07 +02:00
Michael Peter Christen
4058369288 fixed query expressions for collection selection (added quotes) 2013-05-13 13:27:01 +02:00
Michael Peter Christen
f2e36fbd06 enhanced deletion process for very large number of documents 2013-05-13 13:26:24 +02:00
reger
79401cb938 added reindex option for documents with disabled or obsolete fields to Solr Schema Editor page (IndexSchema_p.html) 
this allows to remove obsolete fields from the index (according to current schema config)
by selecting all documents containig disabled fields.
 2013-05-13 04:06:57 +02:00
orbiter
cf36c1614f prevent that concurrent deletion process causes wrong double-check in 
crawl start
 2013-05-12 21:37:45 +02:00
orbiter
aeff31cd44 fix for workflow processor (cause: latest redesign for less threads) 2013-05-12 21:36:20 +02:00
Michael Peter Christen
77faeada4d small memory leak patch 2013-05-11 11:19:06 +02:00
Michael Peter Christen
b24d1d18e4 removed synchronization and concurrency in Fulltext class, concurrent 
deletions are now handled in ConcurrentUpdateSolrConnector
 2013-05-11 10:53:12 +02:00