Marown/yacy_search_server
1
0
Fork 0
mirror of https://github.com/yacy/yacy_search_server.git synced 2024-09-21 00:00:13 +02:00
Code Issues Actions Packages Projects Releases Wiki Activity
a5707cd2eb
yacy_search_server/source/net/yacy/crawler/robots
History
Michael Peter Christen 06afb568e2 new Strategies in Balancer: 
- doublecheck cache now records the crawl depth as well
- doublecheck cache is available from the outside (made static)
- no more need to crawl hosts with lowest depth first, instead all hosts
which have only singleton entries are preferred to reduce the number of
files.
2014-04-17 12:52:54 +02:00
..
RobotsTxt.java new Strategies in Balancer: 2014-04-17 12:52:54 +02:00
RobotsTxtEntry.java support for multiple sitemaps in robots.txt 2014-03-14 13:33:23 +01:00
RobotsTxtParser.java support for multiple sitemaps in robots.txt 2014-03-14 13:33:23 +01:00