Marown/yacy_search_server
1
0
Fork 0
mirror of https://github.com/yacy/yacy_search_server.git synced 2024-09-21 00:00:13 +02:00
Code Issues Actions Packages Projects Releases Wiki Activity
2de159719b
yacy_search_server/source/net/yacy/crawler/data
History
Michael Peter Christen 2de159719b added an option to set 'obey nofollow' for links with rel="nofollow" 
attribute in the <a> tag for each crawl. This introduces a lot of
changes because it extends the usage of the AnchorURL Object type which
now also has a different toString method that the underlying
DigestURL.toString. It is therefore not advised to use .toString at all
for urls, just just toNormalform(false) instead.
2014-07-18 12:43:01 +02:00
..
Cache.java added an option to set 'obey nofollow' for links with rel="nofollow" 2014-07-18 12:43:01 +02:00
CrawlProfile.java added an option to set 'obey nofollow' for links with rel="nofollow" 2014-07-18 12:43:01 +02:00
CrawlQueues.java added an option to set 'obey nofollow' for links with rel="nofollow" 2014-07-18 12:43:01 +02:00
Latency.java
NoticedURL.java made number of open files in crawler configurable and increased default 2014-05-31 09:29:55 +02:00
ResultImages.java
ResultURLs.java