Marown/yacy_search_server
1
0
Fork 0
mirror of https://github.com/yacy/yacy_search_server.git synced 2024-09-21 00:00:13 +02:00
Code Issues Actions Packages Projects Releases Wiki Activity
001e05bb80
yacy_search_server/source/net/yacy/crawler/robots
History
Michael Peter Christen 001e05bb80 do not store failure of loading of robots.txt into the index as a fail 
document
2014-08-01 12:15:14 +02:00
..
RobotsTxt.java do not store failure of loading of robots.txt into the index as a fail 2014-08-01 12:15:14 +02:00
RobotsTxtEntry.java
RobotsTxtParser.java