Marown/yacy_search_server
1
0
Fork 0
mirror of https://github.com/yacy/yacy_search_server.git synced 2024-09-21 00:00:13 +02:00
Code Issues Actions Packages Projects Releases Wiki Activity
96c32e87b0
yacy_search_server/source/de/anomic/crawler
History
orbiter 96c32e87b0 fixes to crawler and new user-agent crawl-delay handling 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7640 6c8d7289-2bf4-0310-a012-ef5d649a1542
2011-04-04 09:47:18 +00:00
..
retrieval fixes to crawler and new user-agent crawl-delay handling 2011-04-04 09:47:18 +00:00
Balancer.java added a handling of appearances of yacy bot entries in robots.txt if this entry addresses the yacy peer 2011-04-03 23:39:45 +00:00
CrawlProfile.java enhancements to web cache and less strict caching rules 2011-03-22 10:35:26 +00:00
CrawlQueues.java added a handling of appearances of yacy bot entries in robots.txt if this entry addresses the yacy peer 2011-04-03 23:39:45 +00:00
CrawlStacker.java - removed file upload function in crawl start and replaced it with an input field for a file path where the crawl start file is loaded. This was necessary to support the API steering for file crawl starts, for two reasons: 2011-03-09 12:50:39 +00:00
CrawlSwitchboard.java enhancements to web cache and less strict caching rules 2011-03-22 10:35:26 +00:00
ImporterException.java
Latency.java fixes to crawler and new user-agent crawl-delay handling 2011-04-04 09:47:18 +00:00
NoticedURL.java added a handling of appearances of yacy bot entries in robots.txt if this entry addresses the yacy peer 2011-04-03 23:39:45 +00:00
ResourceObserver.java same units for memory observer configuration (MiB) 2011-01-02 20:38:01 +00:00
ResultImages.java more memory protection: auto-flush of caches in case of memory shortage 2011-03-09 16:32:34 +00:00
ResultURLs.java - added geo information parsing to html parser 2011-03-30 00:49:47 +00:00
RobotsEntry.java fixes to crawler and new user-agent crawl-delay handling 2011-04-04 09:47:18 +00:00
robotsParser.java fixes to crawler and new user-agent crawl-delay handling 2011-04-04 09:47:18 +00:00
RobotsTxt.java fixes to crawler and new user-agent crawl-delay handling 2011-04-04 09:47:18 +00:00
RSSLoader.java *) set SVN properties 2011-03-08 01:51:51 +00:00
SitemapImporter.java enhanced crawler: 2010-12-11 00:31:57 +00:00
ZURL.java moved getBytes() to UTF8.getBytes() to use a default String encoding 2011-03-10 12:35:32 +00:00