1180 Commits

Author	SHA1	Message	Date
borg-0300	8865948e4e	Cleanup; ... Methode replaceRegex added; Constant MAXPATHLENGTH added; git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1923 6c8d7289-2bf4-0310-a012-ef5d649a1542	2006-03-18 13:34:32 +00:00
orbiter	6c70f4a0cf	renamed wordHashes for a word hash set generation to wordHashSet ... This was done because the wordHashes iterator will get another integer parameter and then conflicts with the wordHashes set generation git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1921 6c8d7289-2bf4-0310-a012-ef5d649a1542	2006-03-18 01:04:16 +00:00
orbiter	d5f8f40c31	removed correcting iterator ... git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1920 6c8d7289-2bf4-0310-a012-ef5d649a1542	2006-03-18 01:01:00 +00:00
orbiter	526407f32e	adoptions, fixes for last commit ... git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1919 6c8d7289-2bf4-0310-a012-ef5d649a1542	2006-03-18 00:21:34 +00:00
orbiter	488a0ed580	replaced old keyIterator and rowIterator by buffered iterators ... that are synchronized with database access Main change is done in kelondroTree, other classes are only adoptions git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1918 6c8d7289-2bf4-0310-a012-ef5d649a1542	2006-03-17 23:43:24 +00:00
hermens	4e9a8f41fd	rwiDBCleaner + dbImporter: Iterate over small excerpts of ... word hashes instead of the whole DB especially while changing the DB in the process. see http://www.yacy-forum.de/viewtopic.php?p=19136#19136 git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1917 6c8d7289-2bf4-0310-a012-ef5d649a1542	2006-03-17 23:39:10 +00:00
hermens	474379ae63	remove TABs from plasmaDbImporter.java ... git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1916 6c8d7289-2bf4-0310-a012-ef5d649a1542	2006-03-17 21:52:36 +00:00
orbiter	dba02f399f	starting of re-design of kelondroTree iterator ... - new access to iterator - added many IOException handling in other Classes git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1914 6c8d7289-2bf4-0310-a012-ef5d649a1542	2006-03-17 20:52:43 +00:00
orbiter	f02b426073	made kelondroTree.nodeIterator private ... git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1910 6c8d7289-2bf4-0310-a012-ef5d649a1542	2006-03-17 18:10:48 +00:00
borg-0300	5f6fdf1786	Bugfix for getCachePath(URL url) ... git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1909 6c8d7289-2bf4-0310-a012-ef5d649a1542	2006-03-17 16:39:29 +00:00
orbiter	303b6463a8	added debug line to URL storage for testing ... see http://www.yacy-forum.de/viewtopic.php?p=19129#19129 git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1908 6c8d7289-2bf4-0310-a012-ef5d649a1542	2006-03-17 16:30:31 +00:00
orbiter	431a4f3609	eliminated correcting iterator in kelondroTree ... VERY EXPERIMENTAL! NOT TESTED! git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1907 6c8d7289-2bf4-0310-a012-ef5d649a1542	2006-03-17 14:20:04 +00:00
orbiter	91dca2cd8d	fixed a bug in last commit: LURL entries cannot be written, ... because a stored property was not set to false (but true) git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1906 6c8d7289-2bf4-0310-a012-ef5d649a1542	2006-03-17 13:07:34 +00:00
orbiter	3286b1f498	re-organisation of lurl-creation and -stacking ... this was necessary to prevent useless write to the database in case of blacklist appearance of the url git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1905 6c8d7289-2bf4-0310-a012-ef5d649a1542	2006-03-17 10:16:07 +00:00
orbiter	9cca36a107	no more strict comparator checking in map exclude method if not needed ... git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1901 6c8d7289-2bf4-0310-a012-ef5d649a1542	2006-03-16 22:46:32 +00:00
orbiter	0b903c5317	removed usage of kelondroNaturalOrder from plasmaCondenser to experimental ... exclude cause of a 100% bug. see http://www.yacy-forum.de/viewtopic.php?p=19076#19076 git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1900 6c8d7289-2bf4-0310-a012-ef5d649a1542	2006-03-16 22:26:04 +00:00
orbiter	4239db0d1c	fixed new ordering for backup iterator TreeSet ... git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1899 6c8d7289-2bf4-0310-a012-ef5d649a1542	2006-03-16 22:20:28 +00:00
borg-0300	399538b7de	Bugfix: wrongly compared ... git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1898 6c8d7289-2bf4-0310-a012-ef5d649a1542	2006-03-16 20:57:06 +00:00
orbiter	33eba5ecb8	temporary disabling last change, does not work (cannot debug right now) ... git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1896 6c8d7289-2bf4-0310-a012-ef5d649a1542	2006-03-16 16:50:56 +00:00
orbiter	f0464042fc	fix for latest iterator-replacement-fix: ... iterator generated TreeSet which did not resprect rotations this has now be implemented using kelondroOrder Objects and by adding this rotation-rules to the ording git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1895 6c8d7289-2bf4-0310-a012-ef5d649a1542	2006-03-16 16:44:29 +00:00
borg-0300	ec21c585cb	try to fix path too long ... see http://www.yacy-forum.de/viewtopic.php?p=19079 git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1893 6c8d7289-2bf4-0310-a012-ef5d649a1542	2006-03-15 20:16:15 +00:00
orbiter	a6a3f4b694	fix for svn 1888 ... this is a redesign of the no-iterator solution git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1892 6c8d7289-2bf4-0310-a012-ef5d649a1542	2006-03-15 16:01:42 +00:00
hydrox	8da13088e9	*)removed multiple DHT_Distribution_Threads ... *)boosted DHT_Distribution sending chunk parallel to multiple peers git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1890 6c8d7289-2bf4-0310-a012-ef5d649a1542	2006-03-15 11:27:43 +00:00
orbiter	283a7181c6	try to fix new 100% cpu bug, possibly caused by iterator method ... see http://www.yacy-forum.de/viewtopic.php?p=18900#18900 git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1888 6c8d7289-2bf4-0310-a012-ef5d649a1542	2006-03-14 23:22:49 +00:00
theli	34c075c1c7	testcommit with subversive ... git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1886 6c8d7289-2bf4-0310-a012-ef5d649a1542	2006-03-14 08:54:44 +00:00
orbiter	f588c0724f	removed cache flush in case of DHT receive ... git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1885 6c8d7289-2bf4-0310-a012-ef5d649a1542	2006-03-13 23:23:45 +00:00
orbiter	e94b374d56	update to cache flush method ... git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1884 6c8d7289-2bf4-0310-a012-ef5d649a1542	2006-03-13 23:06:06 +00:00
orbiter	bcd99fe83e	introduced a second RAM cache for DHT transfer ... git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1880 6c8d7289-2bf4-0310-a012-ef5d649a1542	2006-03-13 10:43:12 +00:00
hydrox	360a460da8	*)URL-Cleaner: moved logging-statement to correct position ... *)plasmaURLPattern: host is now added to the hashset in lowercase git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1879 6c8d7289-2bf4-0310-a012-ef5d649a1542	2006-03-12 18:08:48 +00:00
orbiter	02f9765013	quickfix for time problem during cache restore ... see http://www.yacy-forum.de/viewtopic.php?p=18810#18810 git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1878 6c8d7289-2bf4-0310-a012-ef5d649a1542	2006-03-11 21:55:32 +00:00
orbiter	69d26045eb	quick-fix for http://www.yacy-forum.de/viewtopic.php?p=18805#18805 ... git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1876 6c8d7289-2bf4-0310-a012-ef5d649a1542	2006-03-10 20:45:14 +00:00
allo	b3df5ce2b8	another local iprange ... git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1875 6c8d7289-2bf4-0310-a012-ef5d649a1542	2006-03-10 18:34:53 +00:00
hermens	ad119f06af	*) Don't overwrite new entries with older ones ... see: http://www.yacy-forum.de/viewtopic.php?t=2015 git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1874 6c8d7289-2bf4-0310-a012-ef5d649a1542	2006-03-10 16:28:01 +00:00
orbiter	be88687d8c	fixed some problems with new cache flush karenz ... git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1873 6c8d7289-2bf4-0310-a012-ef5d649a1542	2006-03-10 13:57:30 +00:00
theli	d3da7c9a08	*) Adding support for robots Allow directive ... git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1872 6c8d7289-2bf4-0310-a012-ef5d649a1542	2006-03-09 14:03:54 +00:00
theli	734d18f283	*) more correct robots.txt validation ... - isDisallowed now uses getFile instead of getPath git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1870 6c8d7289-2bf4-0310-a012-ef5d649a1542	2006-03-09 13:04:57 +00:00
hydrox	f046e1814a	*fix or last commit ... git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1869 6c8d7289-2bf4-0310-a012-ef5d649a1542	2006-03-09 12:35:50 +00:00
hydrox	c55c51e2a8	*)added keywords to IndexCleaner_p.java ... *)updated Logging git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1868 6c8d7289-2bf4-0310-a012-ef5d649a1542	2006-03-09 12:23:05 +00:00
theli	f0ad0d2b2b	*) better robots.txt support ... - previously rules for all crawlers and special rules for yacy where combined using AND. Now the general rule will be ignored if there is a special rule for yacy (according to rfc) git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1867 6c8d7289-2bf4-0310-a012-ef5d649a1542	2006-03-09 12:19:46 +00:00
orbiter	ddbeda738e	added minimum age of word in cache to performance menu ... git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1866 6c8d7289-2bf4-0310-a012-ef5d649a1542	2006-03-09 11:31:17 +00:00
orbiter	f188611fc6	apply blacklist on rwis during dht receive ... very experimental! git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1865 6c8d7289-2bf4-0310-a012-ef5d649a1542	2006-03-09 10:46:02 +00:00
theli	915812f597	*) Undoing robots parser policy changes from svn rev. 1421 ... - crawling is not allowed if server returned a 403 statuscode (according to rfc) git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1864 6c8d7289-2bf4-0310-a012-ef5d649a1542	2006-03-09 10:17:37 +00:00
orbiter	0ec28b8f8e	added DBCleaner from Hydrox ... see http://www.yacy-forum.de/viewtopic.php?p=18093#18093 The servlet is now named IndexCleaner_p. See http://localhost:8080/IndexCleaner_p.html The Servlet was adopted to fit in the overall architecture git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1863 6c8d7289-2bf4-0310-a012-ef5d649a1542	2006-03-08 22:06:11 +00:00
theli	5ee0125046	*) adding possibility to configure the server port for seed uploading via scp. ... git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1861 6c8d7289-2bf4-0310-a012-ef5d649a1542	2006-03-08 16:34:05 +00:00
allo	2065138a41	reverted last Commit. ... @Orbiter: kannst du dich dann mal drum kümmern, wenn ich versuche die ganze Sache ans Laufen zu bringen, hast du jedesmal was dagegen. Dann mach du es bitte, du wirst ja wissen was du willst... git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1860 6c8d7289-2bf4-0310-a012-ef5d649a1542	2006-03-08 13:35:38 +00:00
allo	7afa5c1b8e	staticIP fix ... tried to solve http://www.yacy-forum.de/viewtopic.php?p=18663#18663 D 2006/03/08 07:08:20 YACY yacyClient.publishMySeed mySeed error - not proper: IP is not proper: -UNRESOLVED_PATTERN- git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1859 6c8d7289-2bf4-0310-a012-ef5d649a1542	2006-03-08 12:23:26 +00:00
theli	fb4100d47b	*) undoing last commit. ... git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1856 6c8d7289-2bf4-0310-a012-ef5d649a1542	2006-03-08 07:59:32 +00:00
theli	a84cc71218	*) removing getTotalRuntime ... - not needed anymore git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1855 6c8d7289-2bf4-0310-a012-ef5d649a1542	2006-03-08 07:37:21 +00:00
low012	1965a8cf70	*) bugfix for previous commit ... git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1853 6c8d7289-2bf4-0310-a012-ef5d649a1542	2006-03-08 00:38:39 +00:00
low012	1c3b415c35	*) nicer anchors (TOC) ... git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1852 6c8d7289-2bf4-0310-a012-ef5d649a1542	2006-03-08 00:24:49 +00:00