yacy_search_server/source/net/yacy/crawler
Michael Peter Christen 63f58e4785 enhanced strategy in host browser
limit number of fresh hosts in round robin hashes
2020-12-20 23:15:55 +01:00
..
data replaced json library from JSON.org with libandroid-json-java 2020-04-24 11:45:25 +02:00
retrieval increased redirect depth by one 2020-12-20 19:44:16 +01:00
robots Small perf improvement : initialize threads names early when possible 2018-05-23 14:45:35 +02:00
Balancer.java Fixed display of crawler pending URLs counts in HostBrowser.html page. 2017-01-22 12:31:14 +01:00
CrawlStacker.java removes some warning and unused objects 2020-08-03 20:44:31 +02:00
CrawlStarterFromScraper.java Updated a license header typo. 2017-10-30 07:38:47 +01:00
CrawlSwitchboard.java Do not block whole server startup on persisted crawl profile load error 2018-06-19 12:48:17 +02:00
FileCrawlStarterTask.java removed transformer 2018-06-19 00:42:23 +02:00
HarvestProcess.java fix for wrong display of error urls in HostBrowser 2012-12-07 00:31:10 +01:00
HostBalancer.java enhanced strategy in host browser 2020-12-20 23:15:55 +01:00
HostQueue.java Fixed crawl queue folder naming for IPv6 hosts on MS Windows filesystems 2018-08-11 10:02:26 +02:00
IllegalCrawlProfileException.java Crawl from local file : faster task end when manually terminating crawl. 2016-10-22 09:11:20 +02:00
LegacyBalancer.java use supplied url port to get robots.txt in crawlers hostqueue 2016-03-02 00:12:34 +01:00
RecrawlBusyThread.java fixes deleting during recrawl 2020-07-22 22:15:00 +02:00