orbiter a31b9097a4 preparations for mass remote crawls: ... two main changes must be implemented to enable mass remote crawls: - shift control of robots.txt to crawl queue (away from stacker). This is necessary since remote crawls can contain unchecked urls. Each peer must check the robots to prevent that it is misused as crawl agent for unwanted file retrieval - implement new index files that control double-check of remotely crawled urls After removal of robots.txt checking from stacker threads, the multi-threading of this process is void. Multithreading has been removed. Also the thread pools for the crawl threads had been removed, since creation of these threads is not resource-consuming, for a detailed explanation see svn 4106 git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4181 6c8d7289-2bf4-0310-a012-ef5d649a1542		2007-10-29 01:43:20 +00:00
..
seedUpload	fix for seed upload build script	2006-10-09 15:22:20 +00:00
yacyAccessible.java	added keywords, properties;	2006-02-21 00:30:38 +00:00
yacyClient.java	* Complete number localization and provide a more reasonable interface to serverObjects:	2007-10-24 21:38:19 +00:00
yacyCore.java	Small correction	2007-10-20 12:02:31 +00:00
yacyDHTAction.java	re-designed some parts of DHT position calculation (effect is the same as before)	2007-10-01 12:30:23 +00:00
yacyNetwork.java	re-designed some parts of DHT position calculation (effect is the same as before)	2007-10-01 12:30:23 +00:00
yacyNewsAction.java	fix to prevent bad-formed news messages	2007-07-13 09:41:55 +00:00
yacyNewsDB.java	distinct keepOrder strategy, more discrete implementation of enhancement introduced in SVN 4158	2007-10-22 15:26:47 +00:00
yacyNewsPool.java	joined anomic.net.URL, plasmaURL and url hash computation:	2007-09-05 09:01:35 +00:00
yacyNewsQueue.java	- fixed problem with yacyVersion and own version generation	2007-07-16 23:47:21 +00:00
yacyNewsRecord.java	re-designed some parts of DHT position calculation (effect is the same as before)	2007-10-01 12:30:23 +00:00
yacyPeerAction.java	added keywords, properties;	2006-02-21 00:30:38 +00:00
yacyPeerActions.java	preparations for mass remote crawls:	2007-10-29 01:43:20 +00:00
yacySearch.java	re-designed some parts of DHT position calculation (effect is the same as before)	2007-10-01 12:30:23 +00:00
yacySeed.java	seed.CCOUNT is interpreted as a double value not int	2007-10-24 23:25:48 +00:00
yacySeedDB.java	re-designed some parts of DHT position calculation (effect is the same as before)	2007-10-01 12:30:23 +00:00
yacySeedUploader.java	added keywords, properties;	2006-02-21 00:30:38 +00:00
yacyURL.java	preparations for mass remote crawls:	2007-10-29 01:43:20 +00:00
yacyVersion.java	joined anomic.net.URL, plasmaURL and url hash computation:	2007-09-05 09:01:35 +00:00