orbiter a563b05b60 enhanced crawler: ... - added a new queue 'noload' which can be filled with urls where it is already known that the content cannot be loaded. This may be because there is no parser available or the file is too big - the noload queue is emptied with the parser process which indexes the file names only - the 'start from file' functionality now also reads from ftp crawler git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7368 6c8d7289-2bf4-0310-a012-ef5d649a1542		2010-12-11 00:31:57 +00:00
..
yacy	enhanced crawler:	2010-12-11 00:31:57 +00:00