yacy_search_server/source/net/yacy/crawler
Michael Peter Christen 84763126e0 added option to make the YaCy proxy act as the cache is never stale. If
set to 'Always Fresh' the cache is always used if the entry in the cache
exist. This is a good way to archive web content and access it without
going online again in case the documents exist.
To do so, open /Settings_p.html?page=ProxyAccess and check the "Always
Fresh" checkbox.
This is set do false which behave as set before.
If you set this to true, then you have your web archive in DATA/HTCACHE.
Copy this to carry around your private copy of the internet!
2014-11-24 20:28:52 +01:00
..
data reduce number of calls to queue.size() because that may be a bottleneck 2014-11-23 20:09:32 +01:00
retrieval added option to make the YaCy proxy act as the cache is never stale. If 2014-11-24 20:28:52 +01:00
robots more ipv6 bugfixes 2014-10-08 15:21:49 +02:00
Balancer.java - added a new Crawler Balancer: HostBalancer and HostQueues: 2014-04-16 21:34:28 +02:00
CrawlStacker.java explain crawl denial when not switched to intranet mode 2014-10-11 09:02:12 +02:00
CrawlSwitchboard.java added option to reverse-sort YaCy tables (internal API change only) 2014-09-18 11:11:09 +02:00
HarvestProcess.java fix for wrong display of error urls in HostBrowser 2012-12-07 00:31:10 +01:00
HostBalancer.java reduce number of calls to queue.size() because that may be a bottleneck 2014-11-23 20:09:32 +01:00
HostQueue.java more stacks shall be considered for on-demand loading, not only 2014-11-23 20:11:23 +01:00
LegacyBalancer.java special strategy for balancer: do not remove targets with zero wait time 2014-04-18 06:50:07 +02:00