Michael Peter Christen 2de159719b added an option to set 'obey nofollow' for links with rel="nofollow" ... attribute in the <a> tag for each crawl. This introduces a lot of changes because it extends the usage of the AnchorURL Object type which now also has a different toString method that the underlying DigestURL.toString. It is therefore not advised to use .toString at all for urls, just just toNormalform(false) instead.		2014-07-18 12:43:01 +02:00
..
Blacklist.java	refactore URIMetadataNode to further unify interaction with index	2014-04-20 01:41:30 +02:00
BlacklistFile.java
BlacklistHelper.java	First draft of a blacklist API.	2014-04-30 00:48:38 +02:00
FilterEngine.java	- the webgraph shall store all links which appear on a web page and not	2013-09-15 00:30:23 +02:00
LoaderDispatcher.java	added an option to set 'obey nofollow' for links with rel="nofollow"	2014-07-18 12:43:01 +02:00
RegexHelper.java	avoid NPE in regex checker	2013-04-20 10:53:49 +02:00