Marown/yacy_search_server
1
0
Fork 0
mirror of https://github.com/yacy/yacy_search_server.git synced 2024-09-21 00:00:13 +02:00
Code Issues Actions Packages Projects Releases Wiki Activity
ee3820c9cc
yacy_search_server/source/de/anomic/crawler
History
orbiter 461a2a6ec7 enhanced remote crawling: 
- 300 ppm is default now (but this is switched off by default; if you switch it on you may want more traffic?)
- better timing for busy queue
- better amount of remote url retrieval
- better time-out values
- better tracking of availability of remote crawl urls
- more logging for result of receipt sending

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7159 6c8d7289-2bf4-0310-a012-ef5d649a1542
2010-09-16 09:34:17 +00:00
..
retrieval - code cleanup / added debug line for further investigation in HTTPDemon.parseMultipart 2010-09-14 21:03:50 +00:00
Balancer.java redesign of crawl profiles data structure. target will be: 2010-08-31 15:47:47 +00:00
CrawlProfile.java redesign of crawl profiles data structure. target will be: 2010-08-31 15:47:47 +00:00
CrawlQueues.java enhanced remote crawling: 2010-09-16 09:34:17 +00:00
CrawlStacker.java redesign of crawl profiles data structure. target will be: 2010-08-31 15:47:47 +00:00
CrawlSwitchboard.java redesign of crawl profiles data structure. target will be: 2010-08-31 15:47:47 +00:00
ImporterException.java
Latency.java preparations to move the HTCache into cora: 2010-08-23 12:32:02 +00:00
NoticedURL.java redesign of crawl profiles data structure. target will be: 2010-08-31 15:47:47 +00:00
ResourceObserver.java change in handling of the all-visible home path for storage in YaCy: 2010-09-02 19:24:22 +00:00
ResultImages.java redesign of parser interface: 2010-06-29 19:20:45 +00:00
ResultURLs.java - more abstraction (HashMap -> Map) 2010-06-01 13:02:11 +00:00
RobotsEntry.java - replaced pdfbox and fontbox version 1.1.0 with 1.2.1 2010-09-07 17:13:47 +00:00
robotsParser.java enhanced computation speed of many replaceAll string operations 2010-09-05 13:19:42 +00:00
RobotsTxt.java counting crawler traffic again: 2010-09-11 15:58:15 +00:00
RSSLoader.java - added nice colors to feed indexing state messages 2010-08-27 11:56:51 +00:00
SitemapImporter.java redesign of the SortStack and SortStore classes: 2010-09-09 15:30:25 +00:00
ZURL.java fixed crawler bug caused by NPE in logging 2010-08-12 01:29:56 +00:00