mirror of
https://github.com/yacy/yacy_search_server.git
synced 2024-09-21 00:00:13 +02:00
038f956821
appeared after the declaration of robots allow/deny for the crawler because the sitemap parser terminated after the allow/deny rules had been found. Now the parser reads the robots.txt until the end to discover also sitemap rules at the end of the file. |
||
---|---|---|
.. | ||
contentcontrol | ||
cora | ||
crawler | ||
data | ||
document | ||
gui | ||
interaction | ||
kelondro | ||
peers | ||
repository | ||
search | ||
server | ||
upnp | ||
utils | ||
visualization | ||
dbtest.java | ||
migration.java | ||
yacy.java | ||
YaCySearchClient.java |