Marown/yacy_search_server
1
0
Fork 0
mirror of https://github.com/yacy/yacy_search_server.git synced 2024-09-21 00:00:13 +02:00
Code Issues Actions Packages Projects Releases Wiki Activity
9,715 Commits 1 Branch 20 Tags 836 MiB
ac1aad5064
Commit Graph

3 Commits

Author SHA1 Message Date
Michael Peter Christen
038f956821 fix for sitemap detection: the sitemap url was not visible if it 
appeared after the declaration of robots allow/deny for the crawler
because the sitemap parser terminated after the allow/deny rules had
been found. Now the parser reads the robots.txt until the end to
discover also sitemap rules at the end of the file.
 2013-05-10 04:56:58 +02:00
Michael Peter Christen
2d9e577ad0 replaced the custom robots.txt loader by the standard http loader 2012-10-28 22:48:11 +01:00
Michael Peter Christen
00c1c777fa refactoring 2012-09-21 15:48:16 +02:00