orbiter b2fe4b7b1a added a handling of appearances of yacy bot entries in robots.txt if this entry addresses the yacy peer ... (directly or indirectly) and it grants a crawl-delay of 0. Then all forced pause mechanisms in YaCy are switched off and the domain is crawled at full speed. crawl delay values can be assigned to either - all yacy peers using the user-agent yacybot - a specific peer with peer name <peer-name>.yacy or - a specific peer with peer hash <peer-hash>.yacyh git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7639 6c8d7289-2bf4-0310-a012-ef5d649a1542		2011-04-03 23:39:45 +00:00
..
getpageinfo_p.java	added a handling of appearances of yacy bot entries in robots.txt if this entry addresses the yacy peer	2011-04-03 23:39:45 +00:00
getpageinfo_p.xml	replaced auto-dom filter with easy-to-understand Site Link-List crawler option	2010-09-30 12:50:34 +00:00
ynetSearch.java	preparations to move the HTCache into cora:	2010-08-23 12:32:02 +00:00
ynetSearch.json
ynetSearch.xml