Marown/yacy_search_server
1
0
Fork 0
mirror of https://github.com/yacy/yacy_search_server.git synced 2024-09-21 00:00:13 +02:00
Code Issues Actions Packages Projects Releases Wiki Activity
e207c41c8e
yacy_search_server/source/de/anomic/crawler
History
orbiter 5ad7f9612b added crawl settings for three new filters for each crawl: 
must-match for IPs (IPs that are known after DNS resolving for each URL in the crawl queue)
must-not-match for IPs
must-match against a list of country codes (allows only loading from hosts that are hostet in given countries)

note: the settings and input environment is there with that commit, but the values are not yet evaluated

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7976 6c8d7289-2bf4-0310-a012-ef5d649a1542
2011-09-27 21:58:18 +00:00
..
retrieval refactoring: 2011-09-25 16:59:06 +00:00
Balancer.java added a concurrent warmin-up of Table data structures. that should speed-up the start-up process but may also cause stronger CPU load at that time. 2011-09-15 10:01:21 +00:00
CrawlProfile.java added crawl settings for three new filters for each crawl: 2011-09-27 21:58:18 +00:00
CrawlQueues.java refactoring: 2011-09-25 16:59:06 +00:00
CrawlStacker.java refactoring: 2011-09-25 16:59:06 +00:00
CrawlSwitchboard.java added crawl settings for three new filters for each crawl: 2011-09-27 21:58:18 +00:00
ImporterException.java
Latency.java refactoring: 2011-09-25 16:59:06 +00:00
NoticedURL.java added a handling of appearances of yacy bot entries in robots.txt if this entry addresses the yacy peer 2011-04-03 23:39:45 +00:00
ResourceObserver.java refactoring: 2011-09-25 16:59:06 +00:00
ResultImages.java - fixed a bug in crawl start with file name (npe in new url) 2011-04-18 16:11:16 +00:00
ResultURLs.java refactoring: moved all score-related classes to new ranking package 2011-08-22 22:37:53 +00:00
RobotsTxt.java - enhanced ybr ranking computation 2011-05-26 10:57:02 +00:00
RobotsTxtEntry.java hack to reduce resource contention caused by massive UTF8 decodings which use java.nio resources: 2011-05-27 08:24:54 +00:00
RobotsTxtParser.java - refactoring of robots 2011-05-02 14:05:51 +00:00
RSSLoader.java refactoring: 2011-09-25 16:59:06 +00:00
SitemapImporter.java refactoring: 2011-09-25 16:59:06 +00:00
ZURL.java added a concurrent warmin-up of Table data structures. that should speed-up the start-up process but may also cause stronger CPU load at that time. 2011-09-15 10:01:21 +00:00