mirror of
https://github.com/yacy/yacy_search_server.git
synced 2024-09-19 00:01:41 +02:00
fcf6b16db4
New "Media Type detection" section in the advanced crawl start page allow to choose between : - not loading URLs with unknown or unsupported file extension without checking the actual Media Type (relying Content-Type header for now). This was the old default behavior, faster, but not really accurate. - always cross check URL file extension against the actual Media Type. This lets properly parse URLs ending with an apparently odd file extension, but which have actually a supported Media Type such as text/html. Sample URLs with misleading file extensions added as documentation in the crawl start page. fixes issue #244 |
||
|---|---|---|
| .. | ||
| data | ||
| retrieval | ||
| robots | ||
| Balancer.java | ||
| CrawlStacker.java | ||
| CrawlStarterFromScraper.java | ||
| CrawlSwitchboard.java | ||
| FileCrawlStarterTask.java | ||
| HarvestProcess.java | ||
| HostBalancer.java | ||
| HostQueue.java | ||
| IllegalCrawlProfileException.java | ||
| LegacyBalancer.java | ||
| RecrawlBusyThread.java | ||