717 Commits

Author	SHA1	Message	Date
orbiter	c12bb8a6d0	- refactoring of the http client ... - added a protection against memory leaks for the access tracker git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5621 6c8d7289-2bf4-0310-a012-ef5d649a1542	2009-02-19 16:24:46 +00:00
orbiter	62505bb3cb	more bugfixes as recommendet by findbugs ... git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5619 6c8d7289-2bf4-0310-a012-ef5d649a1542	2009-02-17 09:12:47 +00:00
orbiter	94c42691d8	- reject less transmissions as transmission receiver ... - do not flag too much receiver when something goes wrong during transmission as sender git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5616 6c8d7289-2bf4-0310-a012-ef5d649a1542	2009-02-16 21:28:48 +00:00
orbiter	26978b2a25	- better memory protection in kelondro caches: computation of needed memory for cache grow ... - removed excessive gc calls - step to 16 vertical DHT partitions git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5611 6c8d7289-2bf4-0310-a012-ef5d649a1542	2009-02-15 23:35:59 +00:00
orbiter	be0c492ae5	fix for memory leak bug in new dht transmissions ... git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5606 6c8d7289-2bf4-0310-a012-ef5d649a1542	2009-02-14 00:01:05 +00:00
orbiter	40d9849aa4	- better control of chunk size in dht selection ... - more restrict values in selection - step to 4 vertical partitions git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5603 6c8d7289-2bf4-0310-a012-ef5d649a1542	2009-02-12 14:47:32 +00:00
orbiter	411f2212f2	more memory leak fixing hacks ... git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5599 6c8d7289-2bf4-0310-a012-ef5d649a1542	2009-02-11 13:31:10 +00:00
orbiter	985d421f91	found and fixed some memory leaks ... git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5596 6c8d7289-2bf4-0310-a012-ef5d649a1542	2009-02-11 11:24:15 +00:00
orbiter	6c627dbdff	update to the server core ... git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5591 6c8d7289-2bf4-0310-a012-ef5d649a1542	2009-02-10 13:26:26 +00:00
orbiter	5393f356aa	fix for termination problem ... git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5589 6c8d7289-2bf4-0310-a012-ef5d649a1542	2009-02-10 01:08:06 +00:00
orbiter	6a876ecb88	first fixes to the DHT transmission process ... git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5588 6c8d7289-2bf4-0310-a012-ef5d649a1542	2009-02-10 00:48:54 +00:00
orbiter	c25c334b75	replaced old DHT transmission method with new method. Many things have changed! some of them: ... - after a index selection is made, the index is splitted into its vertical components - from differrent index selctions the splitted components can be accumulated before they are placed into the transmission queue - each splitted chunk gets its own transmission thread - multiple transmission threads are started concurrently - the process can be monitored with the blocking queue servlet To implement that, a new package de.anomic.yacy.dht was created. Some old files have been removed. The new index distribution model using a vertical DHT was implemented. An abstraction of this model is implemented in the new dht package as interface. The freeworld network has now a configuration of two vertial partitions; sixteen partitions are planned and will be configured if the process is bug-free. This modification has three main targets: - enhance the DHT transmission speed - with a vertical DHT, a search will speed up. With two partitions, two times. With sixteen, sixteen times. - the vertical DHT will apply a semi-dht for URLs, and peers will receive a fraction of the overall URLs they received before. with two partitions, the fractions will be halve. With sixteen partitions, a 1/16 of the previous number of URLs. BE CAREFULL, THIS IS A MAJOR CODE CHANGE, POSSIBLY FULL OF BUGS AND HARMFUL THINGS. git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5586 6c8d7289-2bf4-0310-a012-ef5d649a1542	2009-02-10 00:06:59 +00:00
orbiter	65a1de6c05	longer timeout for remote crawl queries ... git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5573 6c8d7289-2bf4-0310-a012-ef5d649a1542	2009-02-04 16:55:09 +00:00
orbiter	78b7361937	fixed problem with logging ... YOU MUST DELETE DATA/LOG TO MAKE THIS WORK! (sorry..) git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5552 6c8d7289-2bf4-0310-a012-ef5d649a1542	2009-02-01 00:21:08 +00:00
orbiter	94110df85a	moved logging partially to kelondro ... git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5545 6c8d7289-2bf4-0310-a012-ef5d649a1542	2009-01-31 01:06:56 +00:00
orbiter	024da2916b	refactoring of logging ... git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5544 6c8d7289-2bf4-0310-a012-ef5d649a1542	2009-01-30 23:33:47 +00:00
orbiter	83ce65707a	(almost) completed partition of classes in kelondro ... git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5543 6c8d7289-2bf4-0310-a012-ef5d649a1542	2009-01-30 22:44:20 +00:00
orbiter	7ee494fde5	more refactoring of kelondro: ... - seperated BLOB from table classes - renamed 'coding' package to 'order' git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5542 6c8d7289-2bf4-0310-a012-ef5d649a1542	2009-01-30 22:08:08 +00:00
orbiter	bf93767ec6	refactoring of kelondro database classes ... (to be continued) git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5540 6c8d7289-2bf4-0310-a012-ef5d649a1542	2009-01-30 15:33:00 +00:00
orbiter	fc27bf8c4c	refactoring of kelondro classes: ... kelondro shall become independent from other packages. moved bytebuffer, date and memory to kelondro git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5539 6c8d7289-2bf4-0310-a012-ef5d649a1542	2009-01-30 14:48:11 +00:00
f1ori	aaafe05c02	* revert debug change ... * contains instead of startsWith, because there might me localizied strings * decode punycode for every domainpart seperately (see http://forum.yacy-websuche.de/viewtopic.php?f=9&t=1749) git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5516 6c8d7289-2bf4-0310-a012-ef5d649a1542	2009-01-24 00:33:38 +00:00
orbiter	419469ac27	added more methods to control the vertical DHT (not yet active .. ) ... git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5514 6c8d7289-2bf4-0310-a012-ef5d649a1542	2009-01-23 15:32:27 +00:00
orbiter	dedfc7df7f	removed distinction between DHT-in and DHT-out. This is necessary to make room for the new cell data structure, which cannot use this this distinction in the first place, but will enable the same meaning with different mechanisms (segments, later) ... git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5511 6c8d7289-2bf4-0310-a012-ef5d649a1542	2009-01-22 00:03:54 +00:00
orbiter	5080fc33bf	fix for http://forum.yacy-websuche.de/viewtopic.php?p=12247#p12247 ... git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5506 6c8d7289-2bf4-0310-a012-ef5d649a1542	2009-01-20 22:57:58 +00:00
low012	78778df464	*) this should adjust the Dev/Main detection of the updater to the new version numbers (0.7x is Dev, if x != 0) ... git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5504 6c8d7289-2bf4-0310-a012-ef5d649a1542	2009-01-18 21:48:03 +00:00
orbiter	9d119c6b61	migration of auto-update rules to new release strategy: ... next stable will be 0.7, development releases are 0.*x, experimental will be if x = 1, 2, 3 git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5458 6c8d7289-2bf4-0310-a012-ef5d649a1542	2009-01-09 10:08:11 +00:00
lotus	c8451614f3	fix for overflow ... http://forum.yacy-websuche.de/viewtopic.php?p=11696#p11696 git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5440 6c8d7289-2bf4-0310-a012-ef5d649a1542	2009-01-05 18:28:27 +00:00
orbiter	c4c4c223b9	fixed a problem with attribute flags on RWI entries that prevented proper selection of index-of constraint ... git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5437 6c8d7289-2bf4-0310-a012-ef5d649a1542	2009-01-04 02:27:29 +00:00
orbiter	6072831235	no cr transmission for robinson peers ... see also: http://forum.yacy-websuche.de/viewtopic.php?p=10290#p10290 git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5436 6c8d7289-2bf4-0310-a012-ef5d649a1542	2009-01-03 23:44:42 +00:00
orbiter	e004da48d3	- added fast fingerprint computation for files (any). Will be used in new index dump method ... - refactoring git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5415 6c8d7289-2bf4-0310-a012-ef5d649a1542	2008-12-29 12:22:13 +00:00
f1ori	2d2ce24011	* remove all encoding-stuff from proxy ... encoding is handled by parsers or browser, proxy only passes through git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5410 6c8d7289-2bf4-0310-a012-ef5d649a1542	2008-12-23 19:14:54 +00:00
lotus	449e697436	fix for null-seed in seedfile ... http://forum.yacy-websuche.de/viewtopic.php?f=5&t=1653 git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5401 6c8d7289-2bf4-0310-a012-ef5d649a1542	2008-12-19 12:10:01 +00:00
orbiter	2802138787	- refactoring of CrawlStacker (to prepare it for new multi-Threading to remove DNS lookup bottleneck) ... - fix of shallBeOwnWord target computation heuristic git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5392 6c8d7289-2bf4-0310-a012-ef5d649a1542	2008-12-15 00:02:58 +00:00
orbiter	1779c3c507	- added a read cache to the RAFile interface to RandomAccessFile ... - added a write buffer to BLOBHeap - modified the BLOBBuffer (is now only to buffer non-compressed content) - added content compression to the HTCache The new read cache will decrease the start/initialization time of BLOB files, like the HTCache, RobotsTxt and other BLOBHeap structures. git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5386 6c8d7289-2bf4-0310-a012-ef5d649a1542	2008-12-10 11:15:19 +00:00
orbiter	47292e696a	more performance hacks ... git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5379 6c8d7289-2bf4-0310-a012-ef5d649a1542	2008-12-04 12:54:16 +00:00
orbiter	d39d420b39	performance hacks ... git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5376 6c8d7289-2bf4-0310-a012-ef5d649a1542	2008-12-03 15:38:29 +00:00
orbiter	c6525ab75f	fix for NPE in seed handling ... git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5371 6c8d7289-2bf4-0310-a012-ef5d649a1542	2008-12-01 23:08:27 +00:00
danielr	538359a0ff	simple fix to get DHT working again (maybe something more has to be done ;) ... fix for http://forum.yacy-websuche.de/viewtopic.php?f=6&t=1578 git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5327 6c8d7289-2bf4-0310-a012-ef5d649a1542	2008-11-11 18:55:16 +00:00
f1ori	7e1fe05e3c	* added utf8-encoding to many getBytes-calls ... * utf8 should work now git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5323 6c8d7289-2bf4-0310-a012-ef5d649a1542	2008-11-08 20:24:31 +00:00
orbiter	3f746be5d4	- consolidation and refactoring of many DHT target - computing methods ... - implemented vertical DHT acceptance ("my own DHT") to accept new targets - added new target computation for global search: addresses vertical targets also - enhanced remote crawling: collection of remote crawl urls if queue has less than 100 entries (was: 0 entries) - better performance value computations for PPM selection in network configuration git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5319 6c8d7289-2bf4-0310-a012-ef5d649a1542	2008-11-06 10:07:53 +00:00
orbiter	d014b2728a	Design-check, Extension and Refactoring of DHT target position computation: ... - two different computations (but mathematical equivalent) of the DHT distance had been consolidated - moved from 0.0 .. 1.0 double-range position computation to 0 .. Long.Max range for DHT targets - added fast Long - to - hash computation - high-precision target computation of gaps for new peers - added new target computation for horizontal and vertical DHT targets (not yet in use) - old horizontal-only DHT targets will be upwards compatible to new horizontal and vertical DHT positions git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5318 6c8d7289-2bf4-0310-a012-ef5d649a1542	2008-11-03 00:27:23 +00:00
orbiter	ffed5fc415	fixed problem with lost peers in database ... migrated seedDB from BLOBTree to BLOBHeap git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5263 6c8d7289-2bf4-0310-a012-ef5d649a1542	2008-10-10 14:40:02 +00:00
orbiter	6fb865fbdc	- fix of bug in iterator in kelondroBLOBHeap which caused bug in crawl profile listing ... - some refactoring of classes that use kelondroMap (Map instead of HashMap) git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5262 6c8d7289-2bf4-0310-a012-ef5d649a1542	2008-10-10 08:39:11 +00:00
orbiter	820a03f9d6	- removed some warnings ... - used fix in SVN 5233 for ysearch.java and search.java git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5237 6c8d7289-2bf4-0310-a012-ef5d649a1542	2008-10-01 20:20:39 +00:00
lotus	dda771db9d	- search result layout ... - tray only for windows git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5222 6c8d7289-2bf4-0310-a012-ef5d649a1542	2008-09-29 12:39:57 +00:00
lotus	31c31e54e4	new tray icon image for different icon sizes (e.g. linux) ... git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5216 6c8d7289-2bf4-0310-a012-ef5d649a1542	2008-09-28 08:54:33 +00:00
f1ori	9589dfe080	* removed trayicon popupmenu title ... * added some menu items to trayicon git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5213 6c8d7289-2bf4-0310-a012-ef5d649a1542	2008-09-27 08:25:16 +00:00
lotus	5a637f004d	localized tray ... git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5212 6c8d7289-2bf4-0310-a012-ef5d649a1542	2008-09-26 11:09:54 +00:00
lotus	9d4f0325e1	- removed shutdown from search page (we have it in tray now!) ... - fixed doubleclick action for tray git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5211 6c8d7289-2bf4-0310-a012-ef5d649a1542	2008-09-26 10:55:08 +00:00
lotus	214277dad6	- revert r5202 ... - cleanup - installer checks for JRE 1.6 only git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5210 6c8d7289-2bf4-0310-a012-ef5d649a1542	2008-09-26 06:52:36 +00:00