Marown/yacy_search_server
1
0
Fork 0
mirror of https://github.com/yacy/yacy_search_server.git synced 2024-09-21 00:00:13 +02:00
Code Issues Actions Packages Projects Releases Wiki Activity
c25d7bcb80
yacy_search_server/source/net/yacy/crawler/robots
History
Michael Peter Christen c25d7bcb80 - added concurrency for robots.txt loading 
- changed data model for domain counter
2012-10-29 21:08:45 +01:00
..
RobotsTxt.java - added concurrency for robots.txt loading 2012-10-29 21:08:45 +01:00
RobotsTxtEntry.java
RobotsTxtParser.java replaced the custom robots.txt loader by the standard http loader 2012-10-28 22:48:11 +01:00