orbiter bc96d74813 - clean-up of robots.txt parser ... - added 'yacybot' as key to recognize robots.txt entries for YaCy - removed unused method to get robots.txt from database git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6565 6c8d7289-2bf4-0310-a012-ef5d649a1542		2010-01-11 16:36:30 +00:00
..
retrieval	some patches to get the torrent parser working	2010-01-07 00:42:12 +00:00
AbstractImporter.java	start of a really extensive refactoring which will produce a hierarchical package structure with the domain yacy.net as package root	2009-10-09 23:13:30 +00:00
Balancer.java	applied code changes that are recommended by PMD	2010-01-10 23:09:48 +00:00
CrawlProfile.java	added extensive memory protection logic to avoid out of memory errors that may be caused by the RowCollection memory allocation function	2009-12-09 23:27:26 +00:00
CrawlQueues.java	applied code changes that are recommended by PMD	2010-01-10 23:09:48 +00:00
CrawlStacker.java	applied code changes that are recommended by PMD	2010-01-10 23:09:48 +00:00
CrawlSwitchboard.java	added some modifications recommended by PMD for better performance	2010-01-10 01:40:26 +00:00
Importer.java	refactoring:	2008-05-06 13:44:38 +00:00
ImporterException.java	added final where possible	2008-08-02 12:12:04 +00:00
ImporterManager.java	start of a really extensive refactoring which will produce a hierarchical package structure with the domain yacy.net as package root	2009-10-09 23:13:30 +00:00
Latency.java	- clean-up of robots.txt parser	2010-01-11 16:36:30 +00:00
NoticedURL.java	added extensive memory protection logic to avoid out of memory errors that may be caused by the RowCollection memory allocation function	2009-12-09 23:27:26 +00:00
ResourceObserver.java	* reenable DHT if yet enough memory is available	2010-01-10 19:04:43 +00:00
ResultImages.java	refactoring of yacy documents and parsers: they depend now only on the kelondro classes	2009-10-18 00:53:43 +00:00
ResultURLs.java	preset of proper HashMap dimensions: should prevent re-hashing and increase performance	2009-12-02 14:01:19 +00:00
RobotsEntry.java	applied code changes that are recommended by PMD	2010-01-10 23:09:48 +00:00
robotsParser.java	- clean-up of robots.txt parser	2010-01-11 16:36:30 +00:00
RobotsTxt.java	- clean-up of robots.txt parser	2010-01-11 16:36:30 +00:00
SitemapImporter.java	applied code changes that are recommended by PMD	2010-01-10 23:09:48 +00:00
ZURL.java	applied code changes that are recommended by PMD	2010-01-10 23:09:48 +00:00