orbiter a31b9097a4 preparations for mass remote crawls: ... two main changes must be implemented to enable mass remote crawls: - shift control of robots.txt to crawl queue (away from stacker). This is necessary since remote crawls can contain unchecked urls. Each peer must check the robots to prevent that it is misused as crawl agent for unwanted file retrieval - implement new index files that control double-check of remotely crawled urls After removal of robots.txt checking from stacker threads, the multi-threading of this process is void. Multithreading has been removed. Also the thread pools for the crawl threads had been removed, since creation of these threads is not resource-consuming, for a detailed explanation see svn 4106 git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4181 6c8d7289-2bf4-0310-a012-ef5d649a1542		2007-10-29 01:43:20 +00:00
..
httpc.java	extended client connection clean-up:	2007-10-03 15:06:12 +00:00
httpChunkedInputStream.java	* applied many compiler warning recommendations	2006-08-16 19:49:31 +00:00
httpChunkedOutputStream.java	SSI fix for http://forum.yacy-websuche.de/viewtopic.php?f=6&t=29	2007-06-29 10:45:13 +00:00
httpContentLengthInputStream.java	*) Bugfix: calling close must not close the wrapped input stream, otherwise	2006-10-24 06:09:38 +00:00
httpd.java	* Complete number localization and provide a more reasonable interface to serverObjects:	2007-10-24 21:38:19 +00:00
httpdBoundedSizeOutputStream.java	*) bugfix for wrong proxy traffic accounting	2007-03-16 13:52:48 +00:00
httpdByteCountInputStream.java	*) bugfix for wrong proxy traffic accounting	2007-03-16 13:52:48 +00:00
httpdByteCountOutputStream.java	*) bugfix for wrong proxy traffic accounting	2007-03-16 13:52:48 +00:00
httpdFileHandler.java	try to fix connection problem, possible cause for wrong junior status and non-passive passive peers:	2007-09-23 17:31:29 +00:00
httpdLimitExceededException.java	added option to see ranking scores in surftipps	2006-09-30 23:28:03 +00:00
httpdProxyHandler.java	preparations for mass remote crawls:	2007-10-29 01:43:20 +00:00
httpdRobotsTxtConfig.java	corrected Brausse->Brausze	2007-08-01 22:15:51 +00:00
httpdSwitchboard.java	changed release strategy:	2007-06-16 14:11:52 +00:00
httpHeader.java	joined anomic.net.URL, plasmaURL and url hash computation:	2007-09-05 09:01:35 +00:00
httpRemoteProxyConfig.java	replaced some toString()	2007-02-06 16:26:56 +00:00
httpSSI.java	first version of next-generation search interface:	2007-09-03 23:43:55 +00:00
httpTemplate.java	- redesigned crawl start menu, integrated monitoring pages	2007-09-28 01:21:31 +00:00