Marown/yacy_search_server
1
0
Fork 0
mirror of https://github.com/yacy/yacy_search_server.git synced 2024-09-19 00:01:41 +02:00
Code Issues Actions Packages Projects Releases Wiki Activity
6,848 Commits 1 Branch 20 Tags 836 MiB
10a9cb1971
Commit Graph

6848 Commits

This Branch
This Branch
All Branches
Author SHA1 Message Date
orbiter
10a9cb1971 simplified snippet computation process and separated the algorithm into two classes 
also enhances selection criteria for best snippet line computation

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7182 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2010-09-22 20:50:02 +00:00
lotus
4450c240b7 npe fix http://forum.yacy-websuche.de/viewtopic.php?f=6&t=2982 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7181 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2010-09-22 20:24:07 +00:00
orbiter
84a023cbc8 fixed several search bugs 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7180 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2010-09-21 21:48:42 +00:00
lotus
937dd956d3 save default number of search items via web interface 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7179 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2010-09-21 19:45:49 +00:00
orbiter
4e8cf0c72c added a search box and navigation to api steering servlet 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7178 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2010-09-21 13:10:18 +00:00
orbiter
09c208a3ab patch for corrupted database files (just work on and forget key) 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7177 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2010-09-20 14:38:56 +00:00
orbiter
97ee278931 enhanced search speed: 
- better control of number of running search threads
- no time-out waiting time when no ranking feeding takes place
- local search queries by a remote peer may be faster up to 300 milliseconds
- a local search may even be faster

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7176 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2010-09-20 13:17:25 +00:00
orbiter
ee3820c9cc more logging for strange "java.lang.NoClassDefFoundError: de/anomic/http/server/RequestHeader" error 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7175 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2010-09-20 11:01:44 +00:00
f1ori
b392ca5024 * add option to show YaCy version, usage: 
java -cp lib/yacycore.jar net.yacy.yacy -version

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7174 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2010-09-20 10:21:05 +00:00
orbiter
ac73072924 added a demonstration class: integrate the YaCy search results in own applications 
This class requests a YaCy peer remotely and produces search result objects.
The class was implemented in such a way that it is as short as possible. To get a
better integration of search results, use the cora package.
This class is fully stand-alone, it does not need any other external library other than already contained in JRE.

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7173 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2010-09-20 09:57:36 +00:00
orbiter
377f001e0d sorting of crawl profile names in crawl profile editor, see 
http://forum.yacy-websuche.de/viewtopic.php?p=20851#p20851

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7172 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2010-09-20 09:09:38 +00:00
orbiter
3552476fbe terminated migration from apache httpclient-3.1 to 4.1: 
- remove the library
- added two classes from the httpclient-3.1 library as source code to YaCy because these classes were used by the YaCy HTTP Server
- modified the added classes ChunkedInputStream and ContentLengthInputStream in such a way that:
 * there are no more dependencies to httpclient-3.1
 * these classes had been simplified to serve only the purpose for the YaCy httpd

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7171 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2010-09-20 08:36:48 +00:00
orbiter
8da4eb5de6 addition to patch in SVN 7111 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7170 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2010-09-19 23:12:50 +00:00
orbiter
a2f9974745 some redesign in the access tracker to realize sixcoolers question about "smartes way for deleting the first Object": 
- not so much abstraction for a collection, makes use of remove() (no operands) possible
- different way to delete elements in track (destructive, not constructive (less copies of elements in new queue))
- more abstraction for class api since no static class must be used any more

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7169 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2010-09-19 23:00:24 +00:00
sixcooler
03f0414025 some minor correction of my last commit 
sorry for the noise

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7168 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2010-09-19 20:57:25 +00:00
sixcooler
42fa0eadb1 fix endless loop: 
Collection does not support remove(int)
(isn't there a smartes way for deleting the first Object?)

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7167 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2010-09-19 20:41:44 +00:00
low012
5a9ea0308f *) further simplification of wiki code parser (less redundancy in code, less magic numbers), still not done with it... 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7166 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2010-09-18 11:40:37 +00:00
low012
f32bb5e51f *) Changed image in Steering.html from linked image to embedded image because shutdown is so fast now, browsers can't load image before Yacy instance is gone already. Had to make image smaller since IE does not accept large Base64 encoded images. 
*) Decreases wait time in Steering.html before first check since 
*) HTML fixes

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7165 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2010-09-18 00:18:52 +00:00
orbiter
37baa8bae3 - fixes for concurrency exceptions and failed database integrity verification 
- added link to yacystats peer when peer is more than one day old

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7164 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2010-09-17 10:20:04 +00:00
orbiter
29fe401f93 - some layout and text enhancement for site crawl start 
- Quix0rs patch from http://forum.yacy-websuche.de/viewtopic.php?p=20839#p20839 (parts)

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7163 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2010-09-16 23:00:07 +00:00
orbiter
8c1da27347 - added more comments for user in site crawl servlet 
- added a disable/enable function in case that 'sitemap' is selected for functions that do (not) apply
- better naming of menu items
- limit default crawl depth

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7162 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2010-09-16 22:04:14 +00:00
orbiter
3f958ccc7e fixed localization for renamed expert crawl start servlet 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7161 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2010-09-16 21:29:37 +00:00
orbiter
58b7417a59 - added a new 'easy' crawl start menu which can be used for the special case of loading a complete domain 
- the previous crawl start servet was renamed to CrawlStartExpert_p
- easy crawl start is now default

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7160 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2010-09-16 12:02:43 +00:00
orbiter
461a2a6ec7 enhanced remote crawling: 
- 300 ppm is default now (but this is switched off by default; if you switch it on you may want more traffic?)
- better timing for busy queue
- better amount of remote url retrieval
- better time-out values
- better tracking of availability of remote crawl urls
- more logging for result of receipt sending

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7159 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2010-09-16 09:34:17 +00:00
orbiter
670ba4d52b - removed the remote crawl option from the network configuration submenu and 
- added a remote crawl menu item to the index create menu. This menu also shows a list of peers that provide remote crawl urls
- set remote crawl option by default to off. This option may be important but it also confuses first-time users


git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7158 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2010-09-16 00:39:05 +00:00
orbiter
89c2d8b81e better initial hash computation 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7157 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2010-09-15 22:11:52 +00:00
orbiter
34e2f7f487 enhanced snippet fetch strategy: concurrent snippet fetch even for offline-snippet searches. This improves speed since it is now possible to fetch snippets offline and parsing of source files from the htcache can be enhanced using concurrency. This improves local and remote search. 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7156 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2010-09-15 21:09:14 +00:00
orbiter
0cf006865e refactoring and enhanced concurrency 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7155 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2010-09-15 11:38:03 +00:00
orbiter
83ac07874f - corrected return value of put() methods (not used anywhere, so it did not harm before) 
- added use of LookAheadIterator which should prevent mistakes when coding iterators with embedded iterators
- added a fail-safe reaction in case of database corruption using iterators over database elements (no interruption then)


git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7154 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2010-09-15 10:43:14 +00:00
orbiter
f9a27a05e5 migrated to log4j 1.2.16 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7153 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2010-09-15 09:18:35 +00:00
orbiter
5c67e6ca49 migrated to latest apache commons fileupload 1.2.2 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7152 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2010-09-15 08:54:41 +00:00
orbiter
5702419194 fixed a bug in HTTPClient: keep-alive must be set to false, otherwise servers hold connections 2 seconds open until response. 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7151 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2010-09-14 22:25:35 +00:00
orbiter
5870b13f3a - code cleanup / added debug line for further investigation in HTTPDemon.parseMultipart 
- changed data structure for sorting in search which performs better in that specific case (too many updates)

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7150 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2010-09-14 21:03:50 +00:00
orbiter
ac1c08924e more performance hacks 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7149 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2010-09-14 15:27:27 +00:00
orbiter
14c843d364 more performance hacks 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7148 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2010-09-14 15:00:34 +00:00
orbiter
39f409a7bb performance hacks 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7147 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2010-09-14 14:32:24 +00:00
orbiter
7ebef56add - redesign of a part of the remote search client to make it possible to have a test environment for remote search performance tests 
- added a remote search test main methods in yacyClient

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7146 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2010-09-14 13:35:47 +00:00
orbiter
2e75879504 fix for latest commit 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7145 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2010-09-14 13:01:18 +00:00
orbiter
6e4653cf50 remove DoS protection in remote search for intranet hosts 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7144 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2010-09-14 12:38:05 +00:00
orbiter
3c0e07ba72 removed all delays in shutdown process 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7143 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2010-09-14 09:13:28 +00:00
orbiter
906c572621 - enhanced index create menu structure 
- clear search log caches each time a search is done

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7142 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2010-09-14 09:06:27 +00:00
orbiter
fc924f024e import of oai sources from a list using a command line interface: 
if you have a list of oai servers you can import them all using the linux command:
bin/importOAIList.sh <name-of-oai-list-file>


git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7141 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2010-09-13 10:13:34 +00:00
orbiter
64860dc1bb enhanced search event logging (to be used for further improvements) 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7140 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2010-09-13 09:33:04 +00:00
orbiter
7dbc357593 patch to identify corrupted database files 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7139 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2010-09-13 07:20:53 +00:00
sixcooler
17eebd4ef8 counting crawler traffic again: 
fix for http://forum.yacy-websuche.de/viewtopic.php?f=6&t=2808

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7138 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2010-09-11 15:58:15 +00:00
lotus
547d5226ae fix banner reload parameters (were no html errors) 
adapted default colours

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7137 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2010-09-11 11:05:07 +00:00
lotus
d2a3d08c44 avoid div. by zero 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7136 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2010-09-11 10:58:33 +00:00
orbiter
2c7edea35e - better shutdown behavior for the GUI (waits until data is written if GUI is killed) 
- release 0.97

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7135 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2010-09-10 12:47:24 +00:00
orbiter
34a25856a5 - added navigation to next/prev search page using arrow keys (left/right) 
- better information text for YaCy GUI application

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7134 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2010-09-10 10:42:01 +00:00
orbiter
32f73d1aaa added copy for Info.plist for Mac application release updates (this file contains class paths and start parameters) 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7133 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2010-09-10 09:48:09 +00:00