Marown/yacy_search_server
1
0
Fork 0
mirror of https://github.com/yacy/yacy_search_server.git synced 2024-09-21 00:00:13 +02:00
Code Issues Actions Packages Projects Releases Wiki Activity
5,042 Commits 1 Branch 20 Tags 836 MiB
45ad1c3dd5
Commit Graph

3215 Commits

Author SHA1 Message Date
orbiter
6941bf42b1 performance hacks 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5288 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-10-20 14:07:09 +00:00
orbiter
9b0c4b1063 redesign of parts of the new BLOB buffer 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5287 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-10-19 22:30:44 +00:00
orbiter
1778fb420d - added some performance tweaks to the new BLOB buffer 
- removed the now superfluous HT storage thread
- reduced number of file decompression by shifting the compression moment to the future


git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5286 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-10-19 18:10:42 +00:00
orbiter
9663e61449 added another class to handle BLOB writings to the new HTCACHE data storage: 
- entries are buffered and written as stream with many entries at once (saves many IO accesses)
- entries are compressed with gzip: increases capacity of cache
- concurrency for stream-writing and compression: all writings to the cache are non-blocking

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5284 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-10-18 08:57:48 +00:00
orbiter
382226da94 fix for bug introduced in SVN 5281: parameters were switched 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5282 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-10-17 12:14:57 +00:00
danielr
f2fd043797 refactoring (moved duplicate code into methods) 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5281 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-10-17 10:39:32 +00:00
danielr
c612046e5e r5278 java 1.5 compatible 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5280 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-10-17 09:59:59 +00:00
f1ori
af71ec93bf ops, forgot to import something 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5279 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-10-16 22:44:25 +00:00
f1ori
9e65e9141c * always use UTF-8 for encoding hashes 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5278 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-10-16 22:35:27 +00:00
orbiter
826ca79735 refactoring and new architecture to store the files of the web cache: 
- files are not stored any more as individual files
- a new database structure using BLOBHeap files stores many cache entries in common files
- all file-writing procedures had been migrated to generate byte[] objects which are written with the new database methods

this is only an intermediate step to the final architecture, where cached files are written together with their metadata in one single database structure.

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5276 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-10-16 21:24:09 +00:00
danielr
f095137238 - respecting httpdMaxBusySessions (refusing new connections if limit is hit) 
- comments in serverBusyThread converted to JavaDoc
- better debug output for npe-case in diskUsage


git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5274 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-10-16 10:53:32 +00:00
orbiter
8ba33f104e fix for npe 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5269 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-10-13 21:59:53 +00:00
orbiter
998861acfd - some refactoring in BLOBHeap to enable more gap processing functions 
- better gap merging in BLOBHeap
- shrinking of heap file if gap is at end of file when file is closed

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5268 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-10-13 21:15:54 +00:00
lotus
9d50bfd0b3 fix for npe: http://forum.yacy-websuche.de/viewtopic.php?p=10562 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5267 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-10-13 09:09:53 +00:00
orbiter
766cad6e93 enhancement in memory management of BLOB Heap files / merging of deleted entries 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5266 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-10-12 22:15:01 +00:00
orbiter
7860d5d632 fix for bug in seed list management (cause was bad class overloading, only visual effects!) 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5265 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-10-12 19:51:53 +00:00
orbiter
ffed5fc415 fixed problem with lost peers in database 
migrated seedDB from BLOBTree to BLOBHeap

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5263 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-10-10 14:40:02 +00:00
orbiter
6fb865fbdc - fix of bug in iterator in kelondroBLOBHeap which caused bug in crawl profile listing 
- some refactoring of classes that use kelondroMap (Map instead of HashMap)

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5262 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-10-10 08:39:11 +00:00
orbiter
2d65887723 - fix for bug in new profile handling 
- added a new feature in ymageChart (cannot be seen yet, just wait... will be used in profiling chart)

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5261 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-10-09 22:31:43 +00:00
orbiter
ff68f394dd fix for problem with balancer and lost crawl profiles: 
if crawl profile ist lost, no robots.txt is loaded any more

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5258 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-10-08 18:26:36 +00:00
lotus
fb8d9850ea fix for http://forum.yacy-websuche.de/viewtopic.php?f=5&t=1462 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5248 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-10-05 10:03:02 +00:00
lotus
0d1a2f6183 fix for http://forum.yacy-websuche.de/viewtopic.php?f=5&t=1461 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5247 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-10-04 12:36:11 +00:00
orbiter
9ac16f565b - fixed several bugs in database management functions 
- fixed a display bug for the performance graph
- fixed deadlock when initialization of awt happens simultanously
- removed some debugging output

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5245 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-10-03 18:57:02 +00:00
orbiter
820a03f9d6 - removed some warnings 
- used fix in SVN 5233 for ysearch.java and search.java

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5237 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-10-01 20:20:39 +00:00
lotus
fe2792e9ce use accept-language header instead of user agent for language detection 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5235 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-10-01 17:47:11 +00:00
orbiter
c8bdd965ec - larger update time for status page 
- balancer writes cause of robots.txt in log file for crawl delay
- removed log output for forced GC
- smaller RAM flush for RWI cache, should cause more usage of cache and faster crawling

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5228 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-09-30 11:09:46 +00:00
lotus
dda771db9d - search result layout 
- tray only for windows

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5222 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-09-29 12:39:57 +00:00
orbiter
ce4715e305 removed indexing of anchor links and tagging such words as part of urls (that was wrong) 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5219 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-09-28 21:12:26 +00:00
orbiter
ce57de6cb3 - fixed re-setting of DHT Send/Receive settings 
- small change to network grafics: smaller circles / more URLs necessary for full radius; more PPM necessary for full crawling circles
- fixed exclusion search ('-' did not work any more)
- fixed NPE bug when FTP loader wrote to the error-db

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5218 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-09-28 20:01:10 +00:00
lotus
31c31e54e4 new tray icon image for different icon sizes (e.g. linux) 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5216 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-09-28 08:54:33 +00:00
f1ori
9589dfe080 * removed trayicon popupmenu title 
* added some menu items to trayicon


git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5213 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-09-27 08:25:16 +00:00
lotus
5a637f004d localized tray 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5212 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-09-26 11:09:54 +00:00
lotus
9d4f0325e1 - removed shutdown from search page (we have it in tray now!) 
- fixed doubleclick action for tray

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5211 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-09-26 10:55:08 +00:00
lotus
214277dad6 - revert r5202 
- cleanup
- installer checks for JRE 1.6 only

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5210 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-09-26 06:52:36 +00:00
f1ori
7afa084207 * add nativ java trayicon, using reflections 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5209 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-09-25 19:36:49 +00:00
apfelmaennchen
b97ff24b43 bookmarksDB / xbel.xml: 
- added support for folder=/foldername
- it crashes if foldername ends with /

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5207 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-09-24 21:16:13 +00:00
orbiter
6e7d113eac fix for wrong index initialization after network switch 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5203 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-09-23 23:30:25 +00:00
lotus
0a0cc3bf67 added missing classes to build target "run" 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5201 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-09-23 15:54:12 +00:00
orbiter
7b35d54c6c fixed some problems with network switching (was not completely 'clean') 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5200 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-09-23 12:11:19 +00:00
orbiter
f0b42e5a98 fixed NPE 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5199 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-09-22 14:04:38 +00:00
orbiter
8e0de7f180 update to language statistic evaluation: 
- the condenser does not abandon too small words any more before feeding the statistics
- for text indexing no more urls are used to feed the index (this was wrong, but in contrast the indexing of urls for media search is necessary)
- urls are not used any more to feed the statistics

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5197 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-09-21 20:25:47 +00:00
orbiter
1198eeecc7 added language selection to search query: 
- the language can be selected using a LANGUAGE:<language> element in the query line, i.e.:
java LANGUAGE:en
- the language can be selected with a post element in google-style syntax with the 'rl' element:
?lr=lang_en&query=java

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5193 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-09-21 07:28:57 +00:00
orbiter
00c1535f84 added ranking and evaluation of language type in a search 
the wanted language is taken from the browser user-agent string

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5192 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-09-21 00:04:42 +00:00
lotus
a81cb78211 finally some putHTML on htroot/xml/ 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5188 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-09-20 07:55:30 +00:00
orbiter
bfcf9b7aa3 - added language detection using metadata from documents: html and odt documents provide this information 
- metadata and results from statistical analysis are compared and result is printed out as debug lines
- added ranking profile for wanted language
- added class with ISO 639 table, a list of all valid country codes that will be used for the language identification

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5187 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-09-19 22:19:11 +00:00
apfelmaennchen
5e8bd0f29c small fixes to getpageinfo_p.xml and htmlFilterContentScraper.java with respect to keyword extraction 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5185 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-09-19 14:27:44 +00:00
apfelmaennchen
5b2a57bfd0 - /xml/util/getpageinfo_p.xml added <desc> and <lang> tags 
- changed htmlFilterContentScraper.getKeywords() to split either space or comma charater not both

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5183 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-09-18 21:01:23 +00:00
orbiter
e1f67262f7 - added and removed some debugging output 
- fixed a bug with merge method
- patched wrong output of language identification (not fixed, only patched!)

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5181 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-09-18 14:12:15 +00:00
orbiter
ce2a7ed116 integrated language detection classes into condenser environment 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5180 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-09-18 13:12:33 +00:00
orbiter
2b13705839 fixed a mistake in indexing queue processing: documents had been parsed before it was checked if they should be indexed or not. parsing was not necessary for this check, so the check was moved in the queue in front of the document parsing 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5179 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-09-18 11:36:09 +00:00
orbiter
21dbb39afa switched two balancer cases 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5177 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-09-17 22:13:25 +00:00
orbiter
1bbf362cef update to the crawl balancer: better organization and better crawl delay prediction 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5176 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-09-17 21:45:21 +00:00
orbiter
ddcf285499 - fixed a bug in performance setting (did not work with german translation) 
- reduced maximum number of error url references to save some memory (this was actually a small memory leak)

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5174 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-09-16 23:04:24 +00:00
orbiter
0cd0fee546 fixed bug with wrong proxy result enqueueing. See: 
http://forum.yacy-websuche.de/viewtopic.php?p=8130#p8130
- removed the online status property. This influenced the proxy behavior and created some complexity that was not needed because the online status was never used as it was ceated for (offline browsing)
- checked all proxy identification procedures during crawling and enhanced transparency and error checking
- fixed a proxy identification routine that caused the wrong selection of the proxy result queue

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5173 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-09-16 21:56:23 +00:00
orbiter
670244849d fix for http://forum.yacy-websuche.de/viewtopic.php?p=9835#p9835 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5164 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-09-15 18:29:37 +00:00
lotus
fd9233244e configurable free disk space via disk.free 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5163 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-09-15 17:33:06 +00:00
orbiter
25a62cdc3f small fixes 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5161 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-09-15 15:11:59 +00:00
lotus
73f233bb11 * set resource observer to 1000MB 
* transparent favicon

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5160 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-09-15 12:41:27 +00:00
orbiter
5fbccfd75e fix for http://forum.yacy-websuche.de/viewtopic.php?f=6&t=1366&p=9348#p9348 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5155 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-09-14 20:10:43 +00:00
orbiter
a28faabfd2 fix for http://forum.yacy-websuche.de/viewtopic.php?f=6&t=1351&p=9242#p9242 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5154 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-09-14 20:03:59 +00:00
apfelmaennchen
7b63c66a08 - bugfix in bookmarksDB.Tag.hasPublicItems() 
- this anoying little bug prevented display of public items without admin login for /xml/bookmarks/...

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5151 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-09-14 18:45:08 +00:00
orbiter
1fb1665e71 increased dht interval to avoid peer selection failure 
(maybe too less peers available to fill the big gaps)

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5143 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-09-12 13:38:27 +00:00
orbiter
1eb813bd43 shifted index deletion-on-exit rule to the class where the errors are produced 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5141 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-09-12 11:51:48 +00:00
f1ori
ba76995d2c * fix for http://forum.yacy-websuche.de/viewtopic.php?f=6&t=1415 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5140 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-09-12 10:54:11 +00:00
f1ori
bea6c13139 * with r5137 robotParser didn't work at all -> fix 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5139 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-09-12 09:06:38 +00:00
lotus
3ded1efe84 kelondroExceptionCounter didn't work 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5138 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-09-11 18:51:47 +00:00
f1ori
ae677e1738 * fix problem in robotparser, see http://forum.yacy-websuche.de/viewtopic.php?f=5&t=1421&p=9742 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5137 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-09-11 18:12:17 +00:00
lotus
383d89481e count errors before deleting collection.index 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5136 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-09-10 16:40:20 +00:00
lotus
0bb4fbc403 delete corrupted collecion.index on exit for rebuild on next start 
see http://forum.yacy-websuche.de/viewtopic.php?p=9725#p9725

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5135 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-09-10 12:55:14 +00:00
lotus
b68d06a6e8 performance settings based on network's remote crawl speed 
removed some _pro values from config

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5134 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-09-10 12:52:17 +00:00
danielr
d60b2b198d proxy fixed 'not modified' http://forum.yacy-websuche.de/viewtopic.php?f=6&t=1419 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5133 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-09-10 11:06:22 +00:00
f1ori
bd0318ba81 * YaCy only supports gzip-encoding, so remove any other encoding from request 
* fixes http://www.yacy-forum.org/viewtopic.php?f=2&t=163


git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5132 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-09-09 14:04:52 +00:00
orbiter
bb5c898441 enhancements to localsearch behavior 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5131 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-09-09 10:24:42 +00:00
orbiter
42e2d195ac added hint from http://forum.yacy-websuche.de/viewtopic.php?f=6&t=1294 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5130 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-09-08 22:37:58 +00:00
orbiter
39964e88fa fix for http://forum.yacy-websuche.de/viewtopic.php?f=6&t=1329#p9121 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5129 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-09-08 22:06:45 +00:00
orbiter
3f3673b6e5 extended balancer: 
- added automatic time delay in case that a large number of urls come from the same domain
- added additional time delay in case that an url is a dynamic (CGI) url. This shall cause less IO on targets


git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5128 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-09-08 21:50:37 +00:00
orbiter
3c6e8d2015 set default ppm when network is switched 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5127 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-09-08 18:20:05 +00:00
orbiter
3288c19c1a reduce remote crawl PPM for fresh peers in freeworld to 6 PPM 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5124 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-09-08 09:49:08 +00:00
lotus
5ce9a100bb fix(2) for http://forum.yacy-websuche.de/viewtopic.php?f=6&t=1416 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5122 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-09-07 13:57:53 +00:00
danielr
cf29ca19d4 possible fix for POST character encoding http://forum.yacy-websuche.de/viewtopic.php?f=5&t=1374 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5121 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-09-07 13:10:46 +00:00
danielr
a2eeb6138c fix for http://forum.yacy-websuche.de/viewtopic.php?f=6&t=1416 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5120 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-09-07 13:04:17 +00:00
orbiter
d09ddabd09 corrected a design mistake (5-byte hashes not necessary) 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5119 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-09-04 21:28:00 +00:00
orbiter
c97d0fcee7 modified the domain list export function: 
- used the new superfast domain list generation from the domain statistics
- better interactive behavior

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5118 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-09-04 20:28:36 +00:00
orbiter
77ee0765a4 - added domain statistic generation to IndexControlURLs_p.html servlet 
- added 'delete all' button to all results of such a domain statistic output which causes that all urls to this domain are deleted
- extended stack cleaner to clean also the statistics: they are not completely destroyed, only the smallest counting domains are removed


git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5117 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-09-04 19:41:57 +00:00
orbiter
80a7bc93d6 - added statistical evaluation about domains that appear during crawling 
- added tables that show this statistics in CrawlResults web pages

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5113 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-09-04 09:59:17 +00:00
orbiter
4fbee21cea - added fetch-ahead again (had been removed in last commit) 
- reverted default query mode to verify=false

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5111 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-09-03 23:50:13 +00:00
lotus
423a89ebe8 * fix if yacy was installed to a path with whitespace 
* show nice dots when waiting for restart/update

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5110 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-09-03 18:49:02 +00:00
orbiter
fc03b0437a fixed a error case where a second search after a first search with a different search word failed 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5109 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-09-03 15:55:25 +00:00
orbiter
eca171ba2e fix for case where javascript was not filtered by the html parser 
see http://forum.yacy-websuche.de/viewtopic.php?p=9667#p9667

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5108 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-09-03 14:41:20 +00:00
lotus
e645bae29f display table in log 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5106 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-09-03 13:14:01 +00:00
orbiter
ead39064c5 fixed problem with wrong result number calculation 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5105 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-09-03 10:04:46 +00:00
hermens
2437beb96c fix for http://forum.yacy-websuche.de/viewtopic.php?f=6&t=1360&p=9321#p9321 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5104 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-09-03 07:39:03 +00:00
orbiter
7b12e77a63 fix for http://forum.yacy-websuche.de/viewtopic.php?f=5&t=1393&hilit=&p=9655#p9655 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5103 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-09-03 00:50:42 +00:00
orbiter
05dbba4bab added logging conditions to all fine and finest log line calls 
this will prevent an overhead for the generation of the log lines in case that they then are not printed

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5102 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-09-03 00:30:21 +00:00
orbiter
d3d41e2ee4 - fixed problem with searching with quotes (still not complete, but not as bad as before) 
- fixed parsing of crawl-delay statements when seconds were given with float numbers
- enhanced performance of profiling (not too many loggings; not more than one per second)
- removed some debug output
- fixed wrong return type in logging
- added a logging condition in httpd to prevent that logging statements are generated when they are not written (should be added everywhere!)
- fixed wrong word distance computation in RWI management


git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5101 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-09-02 23:49:48 +00:00
lotus
3fbfd5a78b * fix for non-changing offset on new search term 
* dht-heap doesn't has to be deleted (5097), we simply write a new one on exit
* do not install YaCy in startup because a Windows-shutdown might corrupt something. Installing YaCy as a service would solve this.

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5099 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-09-02 15:09:31 +00:00
danielr
219b93df6a - fixed internal error after receiving chunked POST 
- removed debug output
- added info for "501 Unknown" messages



git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5098 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-08-29 13:51:22 +00:00
lotus
c245c7a45e delete index.dhtin/out.heap if restore fails 
see http://forum.yacy-websuche.de/viewtopic.php?p=9613#p9613

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5097 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-08-29 13:10:41 +00:00
danielr
cd19d0aee6 - added warnings for failed transferRWI (dht-in) 
- fixed parseMultipart (uncompress gzipped body) (dht-in)
- fixed parseMultipart (using content-length only if uncompressed)
- better gzipped POST (chunked instead of content-length) (dht-out)



git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5096 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-08-29 09:42:39 +00:00
orbiter
df4ff423c4 added additional properties to query id's to distinguish search events better 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5093 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-08-28 21:15:59 +00:00