7 Commits

Author	SHA1	Message	Date
Michael Peter Christen	3d4b5e66ce	disallow remote robots to crawl the HostBrowser servlet	2013-11-26 07:06:25 +01:00
cominch	8a91f4fa42	local robots.txt: disallow external crawlers to follow the URL proxy	2012-08-17 11:47:39 +02:00
cominch	6456a1656a	changed local robots.txt to prevent external crawlers to submit random ... search queries	2012-08-16 17:38:10 +02:00
orbiter	0ab6a462ee	- added a missing entry in YaCy interface robots.txt for bookmarks ... - changed default robots.txt deny list to include some more interface pages because the loading of such pages are a peer load issue for YaCy when crawlers come by and information on these pages are not useful for public search. git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7112 6c8d7289-2bf4-0310-a012-ef5d649a1542	2010-09-06 09:58:54 +00:00
karlchenofhell	669f840eab	- added ViewProfile / Impressum (default on) to local peer's robots.txt ... git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3874 6c8d7289-2bf4-0310-a012-ef5d649a1542	2007-06-12 13:48:27 +00:00
karlchenofhell	93e1ad2bca	- fix for last commit ... git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3424 6c8d7289-2bf4-0310-a012-ef5d649a1542	2007-03-02 01:50:21 +00:00
karlchenofhell	88245e44d8	- improved version of robots.txt (delete your old htroot/robots.txt before updating): ... - robots.txt is a servlet now - no need to rewrite the whole file each time a section is added or removed - user-defined disallows, added manually, won't be overwritten anymore - new config-setting: httpd.robots.txt, holding names of the disallowed sections git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3423 6c8d7289-2bf4-0310-a012-ef5d649a1542	2007-03-02 01:19:38 +00:00