Marown/yacy_search_server
1
0
Fork 0
mirror of https://github.com/yacy/yacy_search_server.git synced 2024-09-21 00:00:13 +02:00
Code Issues Actions Packages Projects Releases Wiki Activity
b26787dc2d
yacy_search_server/source/net/yacy/crawler/robots
History
Michael Peter Christen 4948c39e48 added concurrency for mass crawl check
2013-10-23 11:27:19 +02:00
..
RobotsTxt.java added concurrency for mass crawl check 2013-10-23 11:27:19 +02:00
RobotsTxtEntry.java - the webgraph shall store all links which appear on a web page and not 2013-09-15 00:30:23 +02:00
RobotsTxtParser.java - the webgraph shall store all links which appear on a web page and not 2013-09-15 00:30:23 +02:00