orbiter
d34f10c63d
some tests with reverse dns lookup
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2954 6c8d7289-2bf4-0310-a012-ef5d649a1542
2006-11-12 00:28:10 +00:00
orbiter
5a40ea7866
refactoring of wget string list generation
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2692 6c8d7289-2bf4-0310-a012-ef5d649a1542
2006-10-02 09:59:20 +00:00
orbiter
df1629b05a
- code cleanup
...
- version 0.471
- moved surftipps to own web page
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2676 6c8d7289-2bf4-0310-a012-ef5d649a1542
2006-09-29 22:27:20 +00:00
theli
92e986bb91
*) adding missing return prop (requested by allo)
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2532 6c8d7289-2bf4-0310-a012-ef5d649a1542
2006-09-09 12:34:24 +00:00
allo
f0529fe53e
update for ftp urls
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2531 6c8d7289-2bf4-0310-a012-ef5d649a1542
2006-09-09 12:28:28 +00:00
orbiter
3879a0ecd0
replaced java.net.URL usage by use of new class de.anomic.net.URL
...
This shall be seen as an experiment to exclude all cases where
there could be a DNS lookup during URL comparisment.
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2290 6c8d7289-2bf4-0310-a012-ef5d649a1542
2006-07-13 01:21:53 +00:00
allo
62664d7252
AJAX Check for robots.txt before crawling.
...
Icons from herrlich
TODO: Style it nicely ;-)
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1689 6c8d7289-2bf4-0310-a012-ef5d649a1542
2006-02-17 20:55:31 +00:00