4041 Commits

Author	SHA1	Message	Date
orbiter	b88f5fbb4b	slightly changed crawling policy ... git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6723 6c8d7289-2bf4-0310-a012-ef5d649a1542	2010-03-07 01:46:08 +00:00
orbiter	7684a575c4	fix for deletion of error database each time when YaCy starts up ... git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6721 6c8d7289-2bf4-0310-a012-ef5d649a1542	2010-03-07 00:33:39 +00:00
orbiter	f561e340c6	show more results of single domains when not authorized fully (up to 100) ... git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6720 6c8d7289-2bf4-0310-a012-ef5d649a1542	2010-03-07 00:12:58 +00:00
orbiter	c4bdb1e7f2	added one more option in ViewFile to show an iframe like for the orginal web page content but using the cache than the direct link to the content in the web. Upgraded the very old and previously not any more used CacheResource_p servlet to a new and working version. ... git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6719 6c8d7289-2bf4-0310-a012-ef5d649a1542	2010-03-06 23:41:51 +00:00
orbiter	c09a995930	better logging of double occurrences of urls in the crawler ... git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6718 6c8d7289-2bf4-0310-a012-ef5d649a1542	2010-03-06 20:31:30 +00:00
orbiter	884b262130	- added a new Wiki Namespace Navigator ... - some redesign of Navigator data structures git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6716 6c8d7289-2bf4-0310-a012-ef5d649a1542	2010-03-05 21:25:49 +00:00
orbiter	617dfbbd06	allo 'authorization by encoded password' also if requesting client is not from localhost but from the same host as yacy is running on. ... git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6714 6c8d7289-2bf4-0310-a012-ef5d649a1542	2010-03-05 16:03:55 +00:00
orbiter	599c3766c4	added authentication to automated API call ... git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6711 6c8d7289-2bf4-0310-a012-ef5d649a1542	2010-03-04 14:10:03 +00:00
orbiter	727dd9b193	- fixed a bug in robots.txt parser ... - moved storage of robots.txt entries to WorkTables, so it is now possible to browse the robots entries with the table browser git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6710 6c8d7289-2bf4-0310-a012-ef5d649a1542	2010-03-04 11:58:07 +00:00
orbiter	54af9e6b49	- added parsing of robots meta-tag in html headers to detect a noindexing request ... - added evaluation and indexing prevention in case that a noindexing is given in a html file git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6709 6c8d7289-2bf4-0310-a012-ef5d649a1542	2010-03-03 23:32:56 +00:00
orbiter	46c4f8b68a	better look-ahead into the crawl queue: show more on crawl monitor ... git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6699 6c8d7289-2bf4-0310-a012-ef5d649a1542	2010-02-24 23:11:58 +00:00
lotus	7b546415dc	added svn6695 for windows ... git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6697 6c8d7289-2bf4-0310-a012-ef5d649a1542	2010-02-24 14:58:53 +00:00
orbiter	f175f9a2d3	changed way how number of search requests are counted: ... so far only search requests at the remote search interface had been counted. This was done to protect the privacy of searchers, because counting was not done and published at the own search interface. This caused that no search requests of robinson peers had been counted, becuase they cannot be counted at remote peer. This change introduces a distinction of locally done search requests at the local search interface from search requests that are on the local interface but had been submitted from a remote IP without authentication. Now 3 counters are maintained: - partial count of remote searches - total count of local searches on robinson peers from non-authenticated clients - total count of local searches on robinson peers from localhost or authenticated clients In the global statistic of search requests now the first two counters of the three cases are added Because we habe a large number of robinson peers with a large number of remote non-authenticated requests the statistic should show at least three times of the number of search requests. git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6696 6c8d7289-2bf4-0310-a012-ef5d649a1542	2010-02-24 13:53:55 +00:00
orbiter	84222e3b4f	fix for auto-updater: delete old libraries before copy of new one ... git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6695 6c8d7289-2bf4-0310-a012-ef5d649a1542	2010-02-24 13:46:50 +00:00
orbiter	93b7ddc27d	fix for http://forum.yacy-websuche.de/viewtopic.php?p=19376#p19376 ... git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6684 6c8d7289-2bf4-0310-a012-ef5d649a1542	2010-02-21 22:49:35 +00:00
orbiter	8030ed3319	self-healing for lost crawl profile handles ... git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6680 6c8d7289-2bf4-0310-a012-ef5d649a1542	2010-02-18 21:55:45 +00:00
orbiter	e3e5e05ec2	fix for problem in ranking setting which was caused by the introduction of a toString() method in serverObjects ... see also: http://forum.yacy-websuche.de/viewtopic.php?p=19310#p19310 git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6678 6c8d7289-2bf4-0310-a012-ef5d649a1542	2010-02-17 21:31:08 +00:00
orbiter	e3ccfb54aa	fix for display problem in Firefox on MacOS X ... git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6677 6c8d7289-2bf4-0310-a012-ef5d649a1542	2010-02-17 09:08:16 +00:00
orbiter	564927ce72	redesign of CrawlResult data structures because of OOM occurrences during URL deletion processes. ... git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6675 6c8d7289-2bf4-0310-a012-ef5d649a1542	2010-02-16 23:06:04 +00:00
orbiter	ef62d017e5	integrated session id filtering for crawler ... git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6672 6c8d7289-2bf4-0310-a012-ef5d649a1542	2010-02-15 23:15:17 +00:00
orbiter	d8d9984913	added framework for session id filtering (not ready yet) ... git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6671 6c8d7289-2bf4-0310-a012-ef5d649a1542	2010-02-15 22:30:41 +00:00
orbiter	2bc36de336	- fix for bug in svn 6669 ... - cleanup git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6670 6c8d7289-2bf4-0310-a012-ef5d649a1542	2010-02-15 22:06:13 +00:00
orbiter	d378ca4604	better handling of concurrency in seed ... git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6669 6c8d7289-2bf4-0310-a012-ef5d649a1542	2010-02-15 15:57:35 +00:00
orbiter	6538043d89	fix for http://forum.yacy-websuche.de/viewtopic.php?p=19189#p19189 ... git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6668 6c8d7289-2bf4-0310-a012-ef5d649a1542	2010-02-15 15:45:31 +00:00
lotus	945e0ba5a5	allow global search if res. observer disabled index transmission ... git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6658 6c8d7289-2bf4-0310-a012-ef5d649a1542	2010-02-09 17:14:16 +00:00
lotus	8faeedd99a	not a fix! for: ... http://forum.yacy-websuche.de/viewtopic.php?f=5&t=2679 git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6657 6c8d7289-2bf4-0310-a012-ef5d649a1542	2010-02-09 09:33:30 +00:00
lotus	11188cd7eb	resource observer now uses the Java 6 method to check for free space. thus, disk observing now needs Java 6 installed. ... git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6652 6c8d7289-2bf4-0310-a012-ef5d649a1542	2010-02-06 18:48:06 +00:00
orbiter	be18b5d8cd	fix for 'cannot switch back to default language'-bug ... git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6649 6c8d7289-2bf4-0310-a012-ef5d649a1542	2010-02-04 23:53:02 +00:00
orbiter	74e736c903	missing file for last commit ... git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6645 6c8d7289-2bf4-0310-a012-ef5d649a1542	2010-02-04 14:52:58 +00:00
orbiter	308a973503	refactoring of tables data organisation ... git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6644 6c8d7289-2bf4-0310-a012-ef5d649a1542	2010-02-04 11:26:23 +00:00
orbiter	8a76f38d26	Added a new steering servlet that can be used to repeat actions that had been made on the yacy interface. This can be used to: ... - start again a previously started crawl - submit settings (again). This option will be used to transmit all settings of one peer to another peer if the remote-peer steering function is ready This steering framework will also be used for a 'schedule-everything' which will also include a new scheduler for crawling. git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6642 6c8d7289-2bf4-0310-a012-ef5d649a1542	2010-02-03 09:31:12 +00:00
orbiter	840527689b	more simplification of bookmark class ... git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6639 6c8d7289-2bf4-0310-a012-ef5d649a1542	2010-02-01 23:04:52 +00:00
orbiter	d77782a8d5	removed bookmark tags file, tags are now stored only in RAM ... git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6638 6c8d7289-2bf4-0310-a012-ef5d649a1542	2010-02-01 22:44:59 +00:00
orbiter	ada0ce9de3	refactoring of bookmarks: there is a big performance problem in the bookmarks code and furthermore the bookmarks ... will loose its leading role for the re-crawl funtion when the new api tables will work. To be prepared for a replacement of such functions the bookmark class is re-organised. git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6637 6c8d7289-2bf4-0310-a012-ef5d649a1542	2010-02-01 22:18:56 +00:00
orbiter	a131ebbcb5	one more fix for NPE, see ... http://forum.yacy-websuche.de/viewtopic.php?p=19010#p19010 git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6634 6c8d7289-2bf4-0310-a012-ef5d649a1542	2010-02-01 11:28:37 +00:00
orbiter	24060885b6	- added Tables abstraction in data.Tables.java ... fix for http://forum.yacy-websuche.de/viewtopic.php?p=18910#p18910 http://forum.yacy-websuche.de/viewtopic.php?p=18894#p18894 http://forum.yacy-websuche.de/viewtopic.php?p=18814#p18814 git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6631 6c8d7289-2bf4-0310-a012-ef5d649a1542	2010-01-29 18:02:09 +00:00
orbiter	7fdf59a77f	misc NPE check ... git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6630 6c8d7289-2bf4-0310-a012-ef5d649a1542	2010-01-29 15:59:24 +00:00
orbiter	a512aef6ad	fix for http://forum.yacy-websuche.de/viewtopic.php?p=18918#p18918 ... git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6629 6c8d7289-2bf4-0310-a012-ef5d649a1542	2010-01-29 10:33:20 +00:00
orbiter	3889438db6	fix for bookmarks ... git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6615 6c8d7289-2bf4-0310-a012-ef5d649a1542	2010-01-22 14:20:24 +00:00
orbiter	23bcca07a3	removed directly linked servlets that had been there to test memory failures that appeared in that servlets ... git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6612 6c8d7289-2bf4-0310-a012-ef5d649a1542	2010-01-22 13:30:20 +00:00
orbiter	69c29acb6e	no exception thread dump if parser cannot parse becuase that mime-type/extension is in the deny-set ... git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6611 6c8d7289-2bf4-0310-a012-ef5d649a1542	2010-01-22 13:21:37 +00:00
orbiter	8ce936bcdd	added an api recording function: it shall be possible to record ... all operations on YaCy in a database that should make it possible 1) to re-create a setting on fresh peers 2) to transmit a setting from one peer to another 3) to re-create crawl starts after a complete deletion of the index This functionality will also support 4) scheduled re-crawls (new implementation) To implement this, a new database structure has been crated that stores maps into blob heaps. to encode maps the b-encoding technique was used (this is the same encoding that torrent files use) - added a b-encoder - enhanced the b-decoder - added a b-encoded map heap data structure - added a table organisation based on b-encoded heaps - added a servlet to maintain such tables (see Tables_p.html) - integrated the servlet into the Advanced Settings menu - added an api recording based on the new tables git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6606 6c8d7289-2bf4-0310-a012-ef5d649a1542	2010-01-21 22:06:03 +00:00
orbiter	e80e060ca6	- increased thread priority for server threads ... - decreased thread priority for crawler threads git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6596 6c8d7289-2bf4-0310-a012-ef5d649a1542	2010-01-19 11:29:22 +00:00
orbiter	234f733a3d	- relocation of seed db is better for network switch than re-initialization because of the embedding of the peers object in other objects ... - small refactoring of blacklist interface code to remove PMD warnings git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6593 6c8d7289-2bf4-0310-a012-ef5d649a1542	2010-01-18 00:07:20 +00:00
orbiter	473b11033d	fixed network switch process - crawling did not work after a switch before this fix ... git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6592 6c8d7289-2bf4-0310-a012-ef5d649a1542	2010-01-17 23:33:15 +00:00
orbiter	fd7b348973	some fixes for the network switch ... git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6591 6c8d7289-2bf4-0310-a012-ef5d649a1542	2010-01-17 22:07:08 +00:00
orbiter	f6731c6240	more logging etc. ... git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6589 6c8d7289-2bf4-0310-a012-ef5d649a1542	2010-01-17 00:41:50 +00:00
orbiter	d9169cc6c3	increased proxy load time-out from 30000 to 60000 milliseconds ... according to http://forum.yacy-websuche.de/viewtopic.php?p=17782#p17782 git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6583 6c8d7289-2bf4-0310-a012-ef5d649a1542	2010-01-14 10:42:05 +00:00
orbiter	938e806182	tried to fix date problem that may have prevented that foreign peers stay in the network ... - removed unused code - removed possibly wrong utc difference correction git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6581 6c8d7289-2bf4-0310-a012-ef5d649a1542	2010-01-13 20:01:46 +00:00
orbiter	bd05e57d3b	fix for http://forum.yacy-websuche.de/viewtopic.php?p=18563#p18563 ... git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6580 6c8d7289-2bf4-0310-a012-ef5d649a1542	2010-01-13 18:41:33 +00:00