Marown/yacy_search_server
1
0
Fork 0
mirror of https://github.com/yacy/yacy_search_server.git synced 2024-09-19 00:01:41 +02:00
Code Issues Actions Packages Projects Releases Wiki Activity
7baa99f26f
yacy_search_server/source/net/yacy/crawler/robots
History
luccioman 5db1c9155a Do locale independant case conversion on hosts, schemes, and file exts. 
Required for proper operation when the default system locale is Turkish,
as dottless and dotted i characters have specific case conversion rules
in this language.
2017-12-19 13:52:05 +01:00
..
RobotsTxt.java Use unredirected robots.txt URL when adding an entry to the table. 2017-08-16 14:21:07 +02:00
RobotsTxtEntry.java Do locale independant case conversion on hosts, schemes, and file exts. 2017-12-19 13:52:05 +01:00
RobotsTxtParser.java support for multiple sitemaps in robots.txt 2014-03-14 13:33:23 +01:00