luccioman fcf6b16db4 Added new crawler attribute for finer control over Media Type detection ... New "Media Type detection" section in the advanced crawl start page allow to choose between : - not loading URLs with unknown or unsupported file extension without checking the actual Media Type (relying Content-Type header for now). This was the old default behavior, faster, but not really accurate. - always cross check URL file extension against the actual Media Type. This lets properly parse URLs ending with an apparently odd file extension, but which have actually a supported Media Type such as text/html. Sample URLs with misleading file extensions added as documentation in the crawl start page. fixes issue #244		2018-10-25 10:42:12 +02:00
..
net/yacy	Added new crawler attribute for finer control over Media Type detection	2018-10-25 10:42:12 +02:00
log4j.properties