orbiter b2fe4b7b1a added a handling of appearances of yacy bot entries in robots.txt if this entry addresses the yacy peer ... (directly or indirectly) and it grants a crawl-delay of 0. Then all forced pause mechanisms in YaCy are switched off and the domain is crawled at full speed. crawl delay values can be assigned to either - all yacy peers using the user-agent yacybot - a specific peer with peer name <peer-name>.yacy or - a specific peer with peer hash <peer-hash>.yacyh git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7639 6c8d7289-2bf4-0310-a012-ef5d649a1542		2011-04-03 23:39:45 +00:00
..
retrieval	- replaced more direct string-based UTF-8 conversions by predefined UTF-8 conversion	2011-03-10 23:25:07 +00:00
Balancer.java	added a handling of appearances of yacy bot entries in robots.txt if this entry addresses the yacy peer	2011-04-03 23:39:45 +00:00
CrawlProfile.java	enhancements to web cache and less strict caching rules	2011-03-22 10:35:26 +00:00
CrawlQueues.java	added a handling of appearances of yacy bot entries in robots.txt if this entry addresses the yacy peer	2011-04-03 23:39:45 +00:00
CrawlStacker.java	- removed file upload function in crawl start and replaced it with an input field for a file path where the crawl start file is loaded. This was necessary to support the API steering for file crawl starts, for two reasons:	2011-03-09 12:50:39 +00:00
CrawlSwitchboard.java	enhancements to web cache and less strict caching rules	2011-03-22 10:35:26 +00:00
ImporterException.java	added final where possible	2008-08-02 12:12:04 +00:00
Latency.java	added a handling of appearances of yacy bot entries in robots.txt if this entry addresses the yacy peer	2011-04-03 23:39:45 +00:00
NoticedURL.java	added a handling of appearances of yacy bot entries in robots.txt if this entry addresses the yacy peer	2011-04-03 23:39:45 +00:00
ResourceObserver.java	same units for memory observer configuration (MiB)	2011-01-02 20:38:01 +00:00
ResultImages.java	more memory protection: auto-flush of caches in case of memory shortage	2011-03-09 16:32:34 +00:00
ResultURLs.java	- added geo information parsing to html parser	2011-03-30 00:49:47 +00:00
RobotsEntry.java	added a handling of appearances of yacy bot entries in robots.txt if this entry addresses the yacy peer	2011-04-03 23:39:45 +00:00
robotsParser.java	added a handling of appearances of yacy bot entries in robots.txt if this entry addresses the yacy peer	2011-04-03 23:39:45 +00:00
RobotsTxt.java	added a handling of appearances of yacy bot entries in robots.txt if this entry addresses the yacy peer	2011-04-03 23:39:45 +00:00
RSSLoader.java	*) set SVN properties	2011-03-08 01:51:51 +00:00
SitemapImporter.java	enhanced crawler:	2010-12-11 00:31:57 +00:00
ZURL.java	moved getBytes() to UTF8.getBytes() to use a default String encoding	2011-03-10 12:35:32 +00:00