theli
|
2cd695f376
|
*) Bugfix path-entries of robots.txt were not decoded correctly
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@676 6c8d7289-2bf4-0310-a012-ef5d649a1542
|
2005-09-07 11:49:53 +00:00 |
|
theli
|
f8ad65eae1
|
*) First trial implementation of robots.txt support
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@674 6c8d7289-2bf4-0310-a012-ef5d649a1542
|
2005-09-07 11:17:21 +00:00 |
|
allo
|
9300689dde
|
bugfix *gr*
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@662 6c8d7289-2bf4-0310-a012-ef5d649a1542
|
2005-09-05 10:46:11 +00:00 |
|
allo
|
ebc39a7b9a
|
minor fixes
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@659 6c8d7289-2bf4-0310-a012-ef5d649a1542
|
2005-09-05 10:34:26 +00:00 |
|
allo
|
f90f699ab1
|
missing package line.
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@655 6c8d7289-2bf4-0310-a012-ef5d649a1542
|
2005-09-05 10:04:27 +00:00 |
|
allo
|
06a451768f
|
a simple robotsParser.
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@652 6c8d7289-2bf4-0310-a012-ef5d649a1542
|
2005-09-05 07:35:19 +00:00 |
|