orbiter e81be7d4f2 added many missing user-agent declarations for yacy http client connections. ... the most important fix was the addition of the yacybot user-agent for robots.txt loading, because web masters look for that access to see if the crawler behaves correctly. git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4968 6c8d7289-2bf4-0310-a012-ef5d649a1542		2008-07-04 11:03:03 +00:00
..
ftpc.java	- organize imports	2008-06-06 16:01:27 +00:00
natLib.java	added many missing user-agent declarations for yacy http client connections.	2008-07-04 11:03:03 +00:00
whois.java	fix for oom exception during release download	2007-07-03 22:55:47 +00:00