Marown/yacy_search_server
1
0
Fork 0
mirror of https://github.com/yacy/yacy_search_server.git synced 2024-09-21 00:00:13 +02:00
Code Issues Actions Packages Projects Releases Wiki Activity
915812f597
yacy_search_server/source/de
History
theli 915812f597 *) Undoing robots parser policy changes from svn rev. 1421 
- crawling is not allowed if server returned a 403 statuscode (according to rfc)

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1864 6c8d7289-2bf4-0310-a012-ef5d649a1542
2006-03-09 10:17:37 +00:00
..
anomic *) Undoing robots parser policy changes from svn rev. 1421 2006-03-09 10:17:37 +00:00