6 Commits

Author	SHA1	Message	Date
orbiter	f642cfbe30	added hint to the regular expression tester	2014-08-27 18:40:20 +02:00
Michael Peter Christen	2de159719b	added an option to set 'obey nofollow' for links with rel="nofollow" ... attribute in the <a> tag for each crawl. This introduces a lot of changes because it extends the usage of the AnchorURL Object type which now also has a different toString method that the underlying DigestURL.toString. It is therefore not advised to use .toString at all for urls, just just toNormalform(false) instead.	2014-07-18 12:43:01 +02:00
Michael Peter Christen	1b279d7a7e	fixed external link	2014-06-27 15:12:53 +02:00
reger	89e2c5e884	fix: allow enable of CrawlStartExpert.html #file	2014-05-17 22:56:15 +02:00
Michael Peter Christen	a2fba6584f	use submitted default userAgent if cloning a crawl	2014-04-30 05:05:02 +02:00
orbiter	d29b6db270	made crawl start pages public since they do not reveal individual ... information and they are also not used as servlet to actually start the crawl (which is Crawler_p.html).	2014-03-31 20:42:39 +02:00