yacy_search_server/source/de/anomic/crawler
orbiter 474659a71f - modified and enhanced the crawl balancer: better list export, fixing of damaged crawl queue at start-up, re-sorting at start-up to enhance domain order
- added option to set minimum crawl delta for domains in balancer
- added default values to crawl deltas in yacy.init
- added configuration for these deltas in performance queues
- enhanced performance setting computation (more time for indexing queue for a faster flush
- remote crawling is now enabled during local crawling if indexer has space and time for more links
- added database stub for new distributed file system
- refactoring of time computation to get an abstraction level that will be used by a TTL rule in new distributed file system

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4966 6c8d7289-2bf4-0310-a012-ef5d649a1542
2008-07-03 13:08:37 +00:00
..
AbstractImporter.java refactoring: 2008-05-06 13:44:38 +00:00
Balancer.java - modified and enhanced the crawl balancer: better list export, fixing of damaged crawl queue at start-up, re-sorting at start-up to enhance domain order 2008-07-03 13:08:37 +00:00
CrawlEntry.java fix for npe in crawler 2008-05-08 20:16:19 +00:00
CrawlProfile.java - WatchCrawler shows status without JavaScript 2008-06-14 10:24:58 +00:00
CrawlQueues.java - modified and enhanced the crawl balancer: better list export, fixing of damaged crawl queue at start-up, re-sorting at start-up to enhance domain order 2008-07-03 13:08:37 +00:00
CrawlStacker.java some code-cleanup and possible speed enhancements in different core methods 2008-06-17 23:56:39 +00:00
ErrorURL.java refactoring: moved all crawler classes into their own package 2008-05-06 00:32:41 +00:00
FTPLoader.java - added option to switch between yacy networks, especially between the two default networks (freeworld and intranet), 2008-05-14 21:36:02 +00:00
HTTPLoader.java fixed unlimited FileSizeLimit 2008-06-24 19:11:27 +00:00
Importer.java refactoring: 2008-05-06 13:44:38 +00:00
ImporterException.java refactoring: 2008-05-06 13:44:38 +00:00
ImporterManager.java - organize imports 2008-06-06 16:01:27 +00:00
IndexingStack.java - modified and enhanced the crawl balancer: better list export, fixing of damaged crawl queue at start-up, re-sorting at start-up to enhance domain order 2008-07-03 13:08:37 +00:00
LoaderMessage.java refactoring: moved all crawler classes into their own package 2008-05-06 00:32:41 +00:00
NoticedURL.java - modified and enhanced the crawl balancer: better list export, fixing of damaged crawl queue at start-up, re-sorting at start-up to enhance domain order 2008-07-03 13:08:37 +00:00
NoticeURLImporter.java - added option to switch between yacy networks, especially between the two default networks (freeworld and intranet), 2008-05-14 21:36:02 +00:00
ProtocolLoader.java more protection against remote shutdown attacks: prevent loading using the crawler 2008-05-19 23:05:19 +00:00
ResultImages.java refactoring: moved all crawler classes into their own package 2008-05-06 00:32:41 +00:00
ResultURLs.java some code-cleanup and possible speed enhancements in different core methods 2008-06-17 23:56:39 +00:00
RobotsTxt.java refactoring: 2008-06-07 23:12:24 +00:00
SitemapImporter.java refactoring: 2008-05-06 13:44:38 +00:00
ZURL.java some code-cleanup and possible speed enhancements in different core methods 2008-06-17 23:56:39 +00:00