Marown/yacy_search_server
1
0
Fork 0
mirror of https://github.com/yacy/yacy_search_server.git synced 2024-09-21 00:00:13 +02:00
Code Issues Actions Packages Projects Releases Wiki Activity
9,525 Commits 1 Branch 20 Tags 836 MiB
8fb1b1e290
Commit Graph

9525 Commits

This Branch
This Branch
All Branches
Author SHA1 Message Date
Marc Nause
8fb1b1e290 *) simplified banner creation code 2013-05-25 12:56:43 +02:00
Marc Nause
cd0b5f31b4 *) updated links to description of regex 2013-05-25 11:08:06 +02:00
Michael Peter Christen
8f2d3ce2f9 reduced locking situation in crawler: shifted synchronized location and 
reduced time-out of robots.txt load limit
 2013-05-20 22:05:28 +02:00
Michael Peter Christen
f93501e6e0 nice crawl name if crawl is started with file:// (was: null) 2013-05-20 11:25:26 +02:00
Michael Peter Christen
b4f0cac102 added the reindexing job servlet to the submenu structure 2013-05-20 11:02:21 +02:00
reger
97ab5b90e8 - odt & ooxml (office document) parser correction to add content to fulltext index 
- adjust Junit yacyVersionTest & ParserTest 
- update yacyVersion.combined2prettyVersion to the default 4-digit minor ver.
 2013-05-20 01:50:09 +02:00
Michael Peter Christen
b68fbe7d21 Merge branch 'master' of ssh://git@gitorious.org/yacy/rc1.git 
Conflicts:
	source/net/yacy/migration.java
 2013-05-17 14:13:07 +02:00
Michael Peter Christen
06d3063dc9 - no downcase when using collection modifier 
- removed warnings
 2013-05-17 14:11:10 +02:00
Michael Peter Christen
8dbc80da70 redesign of index.exist-test: this shall now not be done using a single 
id to be tested, but with a collection of ids. This will cause only a
single call to solr instead of many. The result is a much better
performace when testing the existence of many urls. The effect should
cause very much less IO during index transmission, both on sender and
receiver side.
 2013-05-17 13:59:37 +02:00
reger
7f63d3747d more generic field selection for reindex option of documents with disabled fields 
using Luke request to compare config with actual fields in index
 2013-05-15 23:16:32 +02:00
Michael Peter Christen
c91c67c3cd reject bad solr requests 2013-05-15 22:42:05 +02:00
Michael Peter Christen
44e363f37f refactoring of WorkflowProcessor, added process counter, update of 
process counter if an blocking thread dies. Added also a new column in
PerformanceConcurrency_p servlet to show the actual number of concurrent
processes.
 2013-05-13 13:28:07 +02:00
Michael Peter Christen
4058369288 fixed query expressions for collection selection (added quotes) 2013-05-13 13:27:01 +02:00
Michael Peter Christen
f2e36fbd06 enhanced deletion process for very large number of documents 2013-05-13 13:26:24 +02:00
reger
79401cb938 added reindex option for documents with disabled or obsolete fields to Solr Schema Editor page (IndexSchema_p.html) 
this allows to remove obsolete fields from the index (according to current schema config)
by selecting all documents containig disabled fields.
 2013-05-13 04:06:57 +02:00
orbiter
cf36c1614f prevent that concurrent deletion process causes wrong double-check in 
crawl start
 2013-05-12 21:37:45 +02:00
orbiter
aeff31cd44 fix for workflow processor (cause: latest redesign for less threads) 2013-05-12 21:36:20 +02:00
Michael Peter Christen
77faeada4d small memory leak patch 2013-05-11 11:19:06 +02:00
Michael Peter Christen
b24d1d18e4 removed synchronization and concurrency in Fulltext class, concurrent 
deletions are now handled in ConcurrentUpdateSolrConnector
 2013-05-11 10:53:12 +02:00
Michael Peter Christen
f965d04496 added new peer icons for Mentor peers and Mentee peers (not used yet) 2013-05-10 17:33:02 +02:00
Michael Peter Christen
b9b446bca6 - added ssl configuration sign (a lock) to network statistic/table 
- fixed a bug in bitfield
 2013-05-10 17:32:21 +02:00
Michael Peter Christen
7095446ad3 added checkbox (near port) to switch on ssl support (https access) to 
the admin interface.
 2013-05-10 13:49:46 +02:00
Michael Peter Christen
e6c8b545c2 Merge branch 'master' of ssh://git@gitorious.org/yacy/rc1.git 2013-05-10 12:16:55 +02:00
orbiter
a83c2fe833 Merge branch 'master' of ssh://git@gitorious.org/yacy/rc1.git 2013-05-10 12:02:40 +02:00
orbiter
4baa0d4a97 Added a default keystore for ssl encryption of the YaCy web interface. 
This will enable https-access to YaCy, but this feature is disabled by
default using the new server.https=false attribute. This has two
purposes:
- make it easier for everyone to use https (just set server.https=true)
- provide the basis for secure yacy-to-yacy communication in the future
 2013-05-10 12:02:31 +02:00
Michael Peter Christen
0aef60f66e Merge branch 'master' of ssh://git@gitorious.org/yacy/rc1.git 2013-05-10 06:03:24 +02:00
reger
da191c839d reduce SolrConnectorLogging setting (from default ALL to INFO) 2013-05-10 05:54:07 +02:00
Michael Peter Christen
aaddb4809c Merge branch 'master' of ssh://git@gitorious.org/yacy/rc1.git 2013-05-10 04:57:15 +02:00
Michael Peter Christen
038f956821 fix for sitemap detection: the sitemap url was not visible if it 
appeared after the declaration of robots allow/deny for the crawler
because the sitemap parser terminated after the allow/deny rules had
been found. Now the parser reads the robots.txt until the end to
discover also sitemap rules at the end of the file.
 2013-05-10 04:56:58 +02:00
reger
4fc6837690 - fix monitor url of crawl job in PerformanceQueues_p.html 
- reduce logging of every index add  (switch embeddedsolr.add from info to debug)
 2013-05-10 04:38:13 +02:00
Michael Peter Christen
442ed50be0 removed some unnecessary synchronizations 2013-05-09 03:06:48 +02:00
Michael Peter Christen
9bd2aee180 migrated to solr 4.3.0 2013-05-09 02:17:53 +02:00
Michael Peter Christen
ad050ec88d - upgraded httpclient, httpcore and httpmime 
- removed httpclient 3.1 which has been used by solrj < 4.x.x and is now
not used any more
- fixed some parts in YaCy which used methods from httpclient 3.1
 2013-05-09 00:22:45 +02:00
Michael Peter Christen
4b100f8b48 Merge branch 'master' of ssh://gitorious.org/yacy/rc1 2013-05-08 23:46:03 +02:00
Michael Peter Christen
3abf516ca7 merged classpath 
Bitte geben Sie eine Versionsbeschreibung für Ihre Änderungen ein. Zeilen,
 2013-05-08 23:45:29 +02:00
orbiter
a1c989002b fix for http://forum.yacy-websuche.de/viewtopic.php?f=5&t=4652 
generate dht data even if dht receive and dht transmission is switched
off
 2013-05-08 16:48:45 +02:00
orbiter
48e9a54e80 updated pdf parser 2013-05-08 15:17:06 +02:00
Michael Peter Christen
e26bdd4a52 fixes to deletion methods (removed unnecessary concurrency and added 
removal of crawl queue entries)
 2013-05-08 13:26:25 +02:00
Michael Peter Christen
f2c9b0b5f2 better robustness of Concurrent Solr Connector against update/deletion 
thread failure
 2013-05-08 12:41:24 +02:00
Michael Peter Christen
f7f3e28c5e prevent that the size of the index is computed too many times. 
Because the index size is now provided by solr, and the only way to do
that is a match for [* TO *], a size computation is quite complex and
time-consuming. Therefore this patch prevents that the method is called
at all and if necessary puts a DOS-preventing barrier in front of it.
 2013-05-08 11:50:46 +02:00
Michael Peter Christen
cca19d94d4 re-declared some fields to be of type string rather than text which 
makes them more efficient and less large
 2013-05-06 16:45:54 +02:00
Michael Peter Christen
cc90f82dbb increased default proxy client timeout to one minute 2013-05-06 14:58:18 +02:00
Michael Peter Christen
ed1d5bace6 draw the names of other peers which receive/send dht into the network 
graphic
 2013-05-06 14:27:39 +02:00
Michael Peter Christen
b528448332 enlarge network graph circle according to image height and reduce the 
image height in the Network servlet. Overall, the image is now larger
but takes less space on the web page.
 2013-05-05 23:39:46 +02:00
Michael Peter Christen
58d85b5b80 Merge branch 'master' of ssh://git@gitorious.org/yacy/rc1.git 2013-05-05 21:05:47 +02:00
reger
24d2b4baee remove pre 1.0 migration statement which possibly overwrites user navigator setting 2013-05-05 05:00:42 +02:00
Michael Peter Christen
f1bb54943e typo 2013-05-04 09:34:06 +02:00
Michael Peter Christen
d7fd346917 - added regular-expression based deletions 
- on-demand collection-list generation for collection-based deletions
instead of a default collection-list presentation (this makes calling
the interface much faster since the computation of collections lists for
large indexes may take some seconds)
 2013-05-04 01:14:10 +02:00
Michael Peter Christen
3841854c97 abstraction of catchall term 2013-05-04 00:14:22 +02:00
Michael Peter Christen
ea85674be2 added the date to error documents 2013-05-04 00:14:00 +02:00