Marown/yacy_search_server
1
0
Fork 0
mirror of https://github.com/yacy/yacy_search_server.git synced 2024-09-19 00:01:41 +02:00
Code Issues Actions Packages Projects Releases Wiki Activity
8,959 Commits 1 Branch 20 Tags 836 MiB
abebb3b124
Commit Graph

1 Commits

Author SHA1 Message Date
Michael Peter Christen
abebb3b124 added a crawl start checker which makes a simple analysis on the list of 
all given urls: shows if the url can be loaded and if there is a robots
and/or a sitemap.
 2012-10-10 02:02:17 +02:00