Marown/yacy_search_server
1
0
Fork 0
mirror of https://github.com/yacy/yacy_search_server.git synced 2024-09-21 00:00:13 +02:00
Code Issues Actions Packages Projects Releases Wiki Activity
9,534 Commits 1 Branch 20 Tags 836 MiB
c2bcfd8afb
Commit Graph

1867 Commits

Author SHA1 Message Date
Michael Peter Christen
c2bcfd8afb Merge branch 'master' of ssh://git@gitorious.org/yacy/rc1.git 2013-05-28 11:39:10 +02:00
Michael Peter Christen
67757b425a use a retry handler with retryCount=0 because we usually expect requests 
to fail if we access non-permanently available resources (peers, web
pages) and want to fail fast without repeating the same request which is
doomed to fail. The previous appearance of http client connection had a
1-2-4-8-second timeout scheme, which caused that connection attempts
lasted for 16 seconds.
 2013-05-28 11:38:45 +02:00
Michael Peter Christen
c2b1075dcf activating pollImmediately in case that DHT receive is off. This will 
cause a much faster search result when running in public robinson mode.
 2013-05-28 10:36:49 +02:00
orbiter
888a985dc6 set a higher limit for table copy usage 2013-05-27 15:23:12 +02:00
Michael Peter Christen
2b563debbf javadoc of new multiple-exist test 2013-05-27 13:45:09 +02:00
Marc Nause
8fb1b1e290 *) simplified banner creation code 2013-05-25 12:56:43 +02:00
Michael Peter Christen
8f2d3ce2f9 reduced locking situation in crawler: shifted synchronized location and 
reduced time-out of robots.txt load limit
 2013-05-20 22:05:28 +02:00
reger
97ab5b90e8 - odt & ooxml (office document) parser correction to add content to fulltext index 
- adjust Junit yacyVersionTest & ParserTest 
- update yacyVersion.combined2prettyVersion to the default 4-digit minor ver.
 2013-05-20 01:50:09 +02:00
Michael Peter Christen
b68fbe7d21 Merge branch 'master' of ssh://git@gitorious.org/yacy/rc1.git 
Conflicts:
	source/net/yacy/migration.java
 2013-05-17 14:13:07 +02:00
Michael Peter Christen
06d3063dc9 - no downcase when using collection modifier 
- removed warnings
 2013-05-17 14:11:10 +02:00
Michael Peter Christen
8dbc80da70 redesign of index.exist-test: this shall now not be done using a single 
id to be tested, but with a collection of ids. This will cause only a
single call to solr instead of many. The result is a much better
performace when testing the existence of many urls. The effect should
cause very much less IO during index transmission, both on sender and
receiver side.
 2013-05-17 13:59:37 +02:00
reger
7f63d3747d more generic field selection for reindex option of documents with disabled fields 
using Luke request to compare config with actual fields in index
 2013-05-15 23:16:32 +02:00
Michael Peter Christen
c91c67c3cd reject bad solr requests 2013-05-15 22:42:05 +02:00
Michael Peter Christen
44e363f37f refactoring of WorkflowProcessor, added process counter, update of 
process counter if an blocking thread dies. Added also a new column in
PerformanceConcurrency_p servlet to show the actual number of concurrent
processes.
 2013-05-13 13:28:07 +02:00
Michael Peter Christen
4058369288 fixed query expressions for collection selection (added quotes) 2013-05-13 13:27:01 +02:00
Michael Peter Christen
f2e36fbd06 enhanced deletion process for very large number of documents 2013-05-13 13:26:24 +02:00
reger
79401cb938 added reindex option for documents with disabled or obsolete fields to Solr Schema Editor page (IndexSchema_p.html) 
this allows to remove obsolete fields from the index (according to current schema config)
by selecting all documents containig disabled fields.
 2013-05-13 04:06:57 +02:00
orbiter
cf36c1614f prevent that concurrent deletion process causes wrong double-check in 
crawl start
 2013-05-12 21:37:45 +02:00
orbiter
aeff31cd44 fix for workflow processor (cause: latest redesign for less threads) 2013-05-12 21:36:20 +02:00
Michael Peter Christen
77faeada4d small memory leak patch 2013-05-11 11:19:06 +02:00
Michael Peter Christen
b24d1d18e4 removed synchronization and concurrency in Fulltext class, concurrent 
deletions are now handled in ConcurrentUpdateSolrConnector
 2013-05-11 10:53:12 +02:00
Michael Peter Christen
b9b446bca6 - added ssl configuration sign (a lock) to network statistic/table 
- fixed a bug in bitfield
 2013-05-10 17:32:21 +02:00
Michael Peter Christen
e6c8b545c2 Merge branch 'master' of ssh://git@gitorious.org/yacy/rc1.git 2013-05-10 12:16:55 +02:00
orbiter
a83c2fe833 Merge branch 'master' of ssh://git@gitorious.org/yacy/rc1.git 2013-05-10 12:02:40 +02:00
orbiter
4baa0d4a97 Added a default keystore for ssl encryption of the YaCy web interface. 
This will enable https-access to YaCy, but this feature is disabled by
default using the new server.https=false attribute. This has two
purposes:
- make it easier for everyone to use https (just set server.https=true)
- provide the basis for secure yacy-to-yacy communication in the future
 2013-05-10 12:02:31 +02:00
Michael Peter Christen
aaddb4809c Merge branch 'master' of ssh://git@gitorious.org/yacy/rc1.git 2013-05-10 04:57:15 +02:00
Michael Peter Christen
038f956821 fix for sitemap detection: the sitemap url was not visible if it 
appeared after the declaration of robots allow/deny for the crawler
because the sitemap parser terminated after the allow/deny rules had
been found. Now the parser reads the robots.txt until the end to
discover also sitemap rules at the end of the file.
 2013-05-10 04:56:58 +02:00
reger
4fc6837690 - fix monitor url of crawl job in PerformanceQueues_p.html 
- reduce logging of every index add  (switch embeddedsolr.add from info to debug)
 2013-05-10 04:38:13 +02:00
Michael Peter Christen
442ed50be0 removed some unnecessary synchronizations 2013-05-09 03:06:48 +02:00
Michael Peter Christen
ad050ec88d - upgraded httpclient, httpcore and httpmime 
- removed httpclient 3.1 which has been used by solrj < 4.x.x and is now
not used any more
- fixed some parts in YaCy which used methods from httpclient 3.1
 2013-05-09 00:22:45 +02:00
orbiter
a1c989002b fix for http://forum.yacy-websuche.de/viewtopic.php?f=5&t=4652 
generate dht data even if dht receive and dht transmission is switched
off
 2013-05-08 16:48:45 +02:00
Michael Peter Christen
e26bdd4a52 fixes to deletion methods (removed unnecessary concurrency and added 
removal of crawl queue entries)
 2013-05-08 13:26:25 +02:00
Michael Peter Christen
f2c9b0b5f2 better robustness of Concurrent Solr Connector against update/deletion 
thread failure
 2013-05-08 12:41:24 +02:00
Michael Peter Christen
f7f3e28c5e prevent that the size of the index is computed too many times. 
Because the index size is now provided by solr, and the only way to do
that is a match for [* TO *], a size computation is quite complex and
time-consuming. Therefore this patch prevents that the method is called
at all and if necessary puts a DOS-preventing barrier in front of it.
 2013-05-08 11:50:46 +02:00
Michael Peter Christen
cca19d94d4 re-declared some fields to be of type string rather than text which 
makes them more efficient and less large
 2013-05-06 16:45:54 +02:00
Michael Peter Christen
cc90f82dbb increased default proxy client timeout to one minute 2013-05-06 14:58:18 +02:00
Michael Peter Christen
ed1d5bace6 draw the names of other peers which receive/send dht into the network 
graphic
 2013-05-06 14:27:39 +02:00
Michael Peter Christen
b528448332 enlarge network graph circle according to image height and reduce the 
image height in the Network servlet. Overall, the image is now larger
but takes less space on the web page.
 2013-05-05 23:39:46 +02:00
reger
24d2b4baee remove pre 1.0 migration statement which possibly overwrites user navigator setting 2013-05-05 05:00:42 +02:00
Michael Peter Christen
3841854c97 abstraction of catchall term 2013-05-04 00:14:22 +02:00
Michael Peter Christen
ea85674be2 added the date to error documents 2013-05-04 00:14:00 +02:00
Michael Peter Christen
6fafed2180 fix for solr cache when a delete buffer is filled and a document, which 
is the delete queue, is replaced with a new one.
 2013-05-03 02:03:30 +02:00
Michael Peter Christen
20b767f35e preventing score computation in solr where applicable 2013-05-03 02:02:35 +02:00
orbiter
7de5b9cfa0 fix for http://bugs.yacy.net/view.php?id=233 
- check geolocation coordinates and accept only those, which are
well-formed
- the solr push process does not stop crawling any more if after 20
requests to Solr Solr does not accept the record. Instead, a severe log
entry asks the user to create a bug request
 2013-05-03 00:24:39 +02:00
Michael Peter Christen
ee217dbdee remove sort order in all cases where not needed 2013-04-30 11:44:56 +02:00
Michael Peter Christen
70e981b333 prevent that long-running deletion tasks block a hard commit. 2013-04-30 11:09:21 +02:00
Michael Peter Christen
bb4bf3d8fd infinity timeout bug protection patch 2013-04-30 11:06:48 +02:00
Michael Peter Christen
1b102d98d8 - added index deletion to index administration submenu 
- added index deletion processes to the process scheduler/recorder
 2013-04-30 02:11:28 +02:00
Michael Peter Christen
d1be4127e7 Merge branch 'master' of ssh://git@gitorious.org/yacy/rc1.git 2013-04-29 19:31:40 +02:00
Michael Peter Christen
1aac722cc6 added another solr connector, the ConcurrentUpdateSolrConnector which 
does not block when long-running updates to solr are made. This is
realized using blocking queues which process all long-running tasks in
the background. Also some bugfixes to existing connectors.
 2013-04-29 19:30:04 +02:00