f1ori 0881190b19 * Robots.txt: don't interpret Crawl-Delays for other robots ... fixes: http://forum.yacy-websuche.de/viewtopic.php?f=5&t=1647 git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5398 6c8d7289-2bf4-0310-a012-ef5d649a1542		2008-12-18 15:35:41 +00:00
..
crawler	* Robots.txt: don't interpret Crawl-Delays for other robots	2008-12-18 15:35:41 +00:00
data	- refactoring of CrawlEntry and CrawlStacker	2008-12-17 22:53:06 +00:00
htmlFilter	speed enhancement for integrated http server:	2008-12-11 20:10:37 +00:00
http	speed enhancement for integrated http server:	2008-12-11 20:10:37 +00:00
icap	more performance hacks	2008-12-04 12:54:16 +00:00
index	- refactoring of CrawlEntry and CrawlStacker	2008-12-17 22:53:06 +00:00
kelondro	- set status of kelondroTree, kelondroBLOBTree and kelondroFlexTable to deprecated	2008-12-18 00:08:17 +00:00
language/identification	integrated language detection classes into condenser environment	2008-09-18 13:12:33 +00:00
net	refactoring and new architecture to store the files of the web cache:	2008-10-16 21:24:09 +00:00
plasma	- set status of kelondroTree, kelondroBLOBTree and kelondroFlexTable to deprecated	2008-12-18 00:08:17 +00:00
server	- refactoring of CrawlEntry and CrawlStacker	2008-12-17 22:53:06 +00:00
tools	more performance hacks	2008-12-04 12:54:16 +00:00
urlRedirector	- refactoring of CrawlEntry and CrawlStacker	2008-12-17 22:53:06 +00:00
xml	more performance hacks	2008-12-04 12:54:16 +00:00
yacy	- refactoring of CrawlStacker (to prepare it for new multi-Threading to remove DNS lookup bottleneck)	2008-12-15 00:02:58 +00:00
ymage	different handling of error cases that occur during loading files with http or ftp:	2008-11-11 21:33:40 +00:00