4149 Commits

Author	SHA1	Message	Date
orbiter	30f108f97d	added stub of oai-pmh importer (not working yet) ... git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6437 6c8d7289-2bf4-0310-a012-ef5d649a1542	2009-10-30 15:58:04 +00:00
orbiter	77c99e500f	added more control over memory allocation ... should avoid some of the OOMs git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6436 6c8d7289-2bf4-0310-a012-ef5d649a1542	2009-10-27 15:25:48 +00:00
orbiter	52470d0de4	- fix for xls parser ... - fix for image parser - temporary integration of images as document types in the crawler and indexer for testing of the image parser git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6435 6c8d7289-2bf4-0310-a012-ef5d649a1542	2009-10-22 22:38:04 +00:00
orbiter	5e8038ac4d	- refactoring of blacklists ... - refactoring of event origin encoding git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6434 6c8d7289-2bf4-0310-a012-ef5d649a1542	2009-10-21 20:14:30 +00:00
orbiter	26fafd85a5	- more refactoring ... - fixed problem with parsers git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6433 6c8d7289-2bf4-0310-a012-ef5d649a1542	2009-10-21 15:12:34 +00:00
orbiter	3528b970d6	- refactoring ... - added new experimental (not-yet-working) image parser - added new test image git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6431 6c8d7289-2bf4-0310-a012-ef5d649a1542	2009-10-19 22:34:44 +00:00
orbiter	a8ce192f63	- shifted main classes to new package net.yacy ... - fixed some bugs in last commit git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6427 6c8d7289-2bf4-0310-a012-ef5d649a1542	2009-10-18 01:38:07 +00:00
orbiter	b79f4f062f	refactoring of yacy documents and parsers: they depend now only on the kelondro classes ... git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6426 6c8d7289-2bf4-0310-a012-ef5d649a1542	2009-10-18 00:53:43 +00:00
hermens	0fd9540866	Configuration of HTTPDProxyHandler logging ... git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6425 6c8d7289-2bf4-0310-a012-ef5d649a1542	2009-10-17 14:04:18 +00:00
orbiter	cee7a05ff2	- de-serialized the pdf parser ... - added fail callback for file indexer git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6415 6c8d7289-2bf4-0310-a012-ef5d649a1542	2009-10-15 10:47:29 +00:00
orbiter	9db928ce53	replaced fontbox 0.7.3 with fontbox 0.8.0 ... git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6414 6c8d7289-2bf4-0310-a012-ef5d649a1542	2009-10-15 09:51:16 +00:00
orbiter	c2272785c7	- fix for xlsx and pptx parsing ... - less exception logging for swf parser git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6413 6c8d7289-2bf4-0310-a012-ef5d649a1542	2009-10-14 19:15:38 +00:00
orbiter	c864901087	- moved httpd.mime to defaults path ... - some documentation fixes - adopted a default setting for the search window: moves css setting to base.css - some enhancements for the DocumentIndex class git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6410 6c8d7289-2bf4-0310-a012-ef5d649a1542	2009-10-14 13:29:09 +00:00
low012	8829ec5f18	*) made sure that   is replaced with a space and not just deleted in CharacterCoding.java ... *) added annotations and made minor changes to serverObjects.java *) set subversion properties for several files git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6409 6c8d7289-2bf4-0310-a012-ef5d649a1542	2009-10-13 20:57:56 +00:00
orbiter	6c347a37eb	more options for DocumentIndex ... git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6408 6c8d7289-2bf4-0310-a012-ef5d649a1542	2009-10-13 08:43:02 +00:00
orbiter	6192205533	more final modifier ... git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6407 6c8d7289-2bf4-0310-a012-ef5d649a1542	2009-10-12 21:59:39 +00:00
orbiter	0f6b011e1a	fix for new index location and better way to use own classes by reflection ... git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6406 6c8d7289-2bf4-0310-a012-ef5d649a1542	2009-10-12 21:12:42 +00:00
orbiter	7a3bbd950f	:-( ... git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6405 6c8d7289-2bf4-0310-a012-ef5d649a1542	2009-10-12 20:29:03 +00:00
orbiter	b953f04f90	one more reflection fix ... git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6404 6c8d7289-2bf4-0310-a012-ef5d649a1542	2009-10-12 17:45:42 +00:00
orbiter	77d6604856	fix for npe, see http://forum.yacy-websuche.de/viewtopic.php?p=17727#p17727 ... git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6403 6c8d7289-2bf4-0310-a012-ef5d649a1542	2009-10-12 17:41:16 +00:00
orbiter	2a7fe35f92	performance tuning using more final modifiers in the kelondro core ... git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6402 6c8d7289-2bf4-0310-a012-ef5d649a1542	2009-10-12 17:37:12 +00:00
orbiter	cb4de9ceee	fixed a bug in table iterator (did not recognize elements in write buffer) ... git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6401 6c8d7289-2bf4-0310-a012-ef5d649a1542	2009-10-12 08:06:35 +00:00
orbiter	e7f18ba24b	refactoring ... git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6399 6c8d7289-2bf4-0310-a012-ef5d649a1542	2009-10-11 00:24:42 +00:00
orbiter	ce8dc575ca	refactoring ... git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6398 6c8d7289-2bf4-0310-a012-ef5d649a1542	2009-10-11 00:12:19 +00:00
orbiter	bea3b99aff	moved table and util classes ... git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6397 6c8d7289-2bf4-0310-a012-ef5d649a1542	2009-10-10 01:14:19 +00:00
orbiter	bd876eb4b7	moved io classes ... git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6396 6c8d7289-2bf4-0310-a012-ef5d649a1542	2009-10-10 01:00:49 +00:00
orbiter	c0e0e1f422	moved blob classes ... git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6395 6c8d7289-2bf4-0310-a012-ef5d649a1542	2009-10-10 00:43:25 +00:00
orbiter	1e4f8b56ed	accumulated classes from different packages into the new rwi package ... git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6394 6c8d7289-2bf4-0310-a012-ef5d649a1542	2009-10-10 00:39:15 +00:00
orbiter	194da25a2f	moved kelondro index ... git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6393 6c8d7289-2bf4-0310-a012-ef5d649a1542	2009-10-09 23:32:08 +00:00
orbiter	4446acc8cd	moved kelondro order ... git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6392 6c8d7289-2bf4-0310-a012-ef5d649a1542	2009-10-09 23:22:22 +00:00
orbiter	f677d534b1	start of a really extensive refactoring which will produce a hierarchical package structure with the domain yacy.net as package root ... - moved here the logging classes as part of the new net.yacy.kelondro package git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6391 6c8d7289-2bf4-0310-a012-ef5d649a1542	2009-10-09 23:13:30 +00:00
orbiter	ea473e32b8	refactoring ... git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6390 6c8d7289-2bf4-0310-a012-ef5d649a1542	2009-10-09 22:27:50 +00:00
orbiter	735e2737e3	* added index segments ... This is a major change in the organization of indexes. Please consider a back-up of your data before you run this update. All existing index files will be moved and renamed to a new position. With this change, it will be possible to maintain different indexes for different purposes and it will be possible to have a distinction between DHT-in and DHT-out specific indexes. Tenants may also have their own index, and it may be possible to have histories and back-ups of indexes. This is just the beginning, many servlets must be adopted after this change, but all functions that had been there should still work. git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6389 6c8d7289-2bf4-0310-a012-ef5d649a1542	2009-10-09 14:44:20 +00:00
orbiter	09de5da74a	once again a performance hack ... git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6388 6c8d7289-2bf4-0310-a012-ef5d649a1542	2009-10-08 18:26:54 +00:00
orbiter	2f6d88403e	uä ... git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6387 6c8d7289-2bf4-0310-a012-ef5d649a1542	2009-10-08 18:10:56 +00:00
orbiter	d2615ea5a8	increased memory for scraper buffer to enhance parsing speed ... git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6386 6c8d7289-2bf4-0310-a012-ef5d649a1542	2009-10-08 15:27:13 +00:00
orbiter	4bbbb74ec4	removed not necessary synchronization ... git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6385 6c8d7289-2bf4-0310-a012-ef5d649a1542	2009-10-08 15:26:28 +00:00
hermens	67e5464cc2	Fix for SVN6380: x[] Arrays are unsuitable Keys for Maps without using a proper Comparator. ... git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6384 6c8d7289-2bf4-0310-a012-ef5d649a1542	2009-10-08 12:55:36 +00:00
hermens	aeab8c7917	Prevent failed DHT attemps from overwriting newer peer info ... git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6382 6c8d7289-2bf4-0310-a012-ef5d649a1542	2009-10-08 00:17:29 +00:00
hermens	9324b5b6c5	Enhancements to DHT ... - speed up deletion of containers when selscted from whole index - correctly eliminate all references to unavailable URLs, not just the first encountered git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6381 6c8d7289-2bf4-0310-a012-ef5d649a1542	2009-10-08 00:03:16 +00:00
hermens	e49e2d75fe	Limit the time Transmission.Chunks stay in the transmissionCloud by using a Map that iterates entires in insertion order. ... git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6380 6c8d7289-2bf4-0310-a012-ef5d649a1542	2009-10-07 23:41:25 +00:00
orbiter	92db7c5d07	increased timeout for index retrieval ... git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6379 6c8d7289-2bf4-0310-a012-ef5d649a1542	2009-10-07 13:03:13 +00:00
lotus	386b9f35f6	activated resource observer for windows 7 ... git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6378 6c8d7289-2bf4-0310-a012-ef5d649a1542	2009-10-07 06:20:24 +00:00
orbiter	6e0dc39a7d	- some fixes to prevent blocking situations ... - better logging for the crawler - better default values for the crawler git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6377 6c8d7289-2bf4-0310-a012-ef5d649a1542	2009-10-06 21:52:55 +00:00
orbiter	51f2bbf04b	possible fix for problem in http://forum.yacy-websuche.de/viewtopic.php?p=17655#p17655 ... git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6376 6c8d7289-2bf4-0310-a012-ef5d649a1542	2009-10-06 09:56:14 +00:00
orbiter	f8371707e5	- possibly better termination for SplitTable ... - better abstraction in DidYouMean git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6375 6c8d7289-2bf4-0310-a012-ef5d649a1542	2009-10-05 22:09:58 +00:00
orbiter	87780f2562	produce did-you-mean also for queries with more than one word ... git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6374 6c8d7289-2bf4-0310-a012-ef5d649a1542	2009-10-05 21:51:02 +00:00
orbiter	04a548a1e3	- temporary integrated the transferURL servlet as static class instead as a class that is called using reflection to investigate the OOM problems in that class ... - fixes for numerous other problems - removed dead code - resdesign of the strings-method, which produces now less memory overhead and may help to prevent OOMs - another fix for the deadlock problem in SplitTable git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6373 6c8d7289-2bf4-0310-a012-ef5d649a1542	2009-10-05 20:11:41 +00:00
orbiter	ea427df944	fixed a worst case situation of the condenser which may cause a temporary full CPU load because of a bad data structure usage ... git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6372 6c8d7289-2bf4-0310-a012-ef5d649a1542	2009-10-05 08:26:55 +00:00
orbiter	3e38035389	fix for interrupted thread during has() property check ... git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6370 6c8d7289-2bf4-0310-a012-ef5d649a1542	2009-10-02 10:55:40 +00:00
orbiter	5bd1c1d205	just added some comments that had been produced to learn about OAI-PMH ... git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6369 6c8d7289-2bf4-0310-a012-ef5d649a1542	2009-10-01 22:56:22 +00:00
orbiter	6aa474f529	- better logging for web cache access and fail reasons ... - better Exception handling for web cache access - distinction between access of web cache for proxy and crawler git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6367 6c8d7289-2bf4-0310-a012-ef5d649a1542	2009-10-01 13:08:19 +00:00
orbiter	3671c37989	added experimental oai-pmh reader and integrated it with the existing dublin core parser ... git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6366 6c8d7289-2bf4-0310-a012-ef5d649a1542	2009-09-30 22:11:00 +00:00
orbiter	58a00205d5	re-activated the emergency close when too many server connections exist ... git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6364 6c8d7289-2bf4-0310-a012-ef5d649a1542	2009-09-30 14:29:43 +00:00
orbiter	c57d2070e6	more logging ... git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6363 6c8d7289-2bf4-0310-a012-ef5d649a1542	2009-09-30 13:25:08 +00:00
orbiter	a995b95367	tried a fix for the httpd access bug (too many unclosed sessions) ... git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6362 6c8d7289-2bf4-0310-a012-ef5d649a1542	2009-09-30 13:18:02 +00:00
orbiter	e1fba41cad	better logging ... git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6361 6c8d7289-2bf4-0310-a012-ef5d649a1542	2009-09-29 21:52:17 +00:00
orbiter	2275f885a8	possible fix for concurrency problem ... git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6360 6c8d7289-2bf4-0310-a012-ef5d649a1542	2009-09-29 21:40:50 +00:00
low012	a6a3090c3d	*) blacklist cleaner supports usage of regular expressions now ... *) refacored BlacklistCleaner_p.java for better readability *) moved check of validity of patterns to the Balcklist implementation since patterns might be valid in one implementation, but not in another *) added method to check validity to Blacklist interface *) fixed some minor issues like typos or wrong whitespaces *) set subversion properties for a whole bunch of files git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6359 6c8d7289-2bf4-0310-a012-ef5d649a1542	2009-09-29 21:28:49 +00:00
orbiter	5a93807781	improved web cache speed: ... - removed one computation out of a synchronization - removed one not necessary has() call git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6358 6c8d7289-2bf4-0310-a012-ef5d649a1542	2009-09-29 08:41:05 +00:00
orbiter	2e8b2867ff	double performance of store method because it avoids one 'has' ... git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6357 6c8d7289-2bf4-0310-a012-ef5d649a1542	2009-09-29 08:23:44 +00:00
orbiter	afda5b1adc	new join method for indexes (not yet used) ... git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6356 6c8d7289-2bf4-0310-a012-ef5d649a1542	2009-09-29 08:16:24 +00:00
orbiter	65b66c2c18	better handling of array files of length 0 ... git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6355 6c8d7289-2bf4-0310-a012-ef5d649a1542	2009-09-29 08:13:44 +00:00
orbiter	1957b5797a	fix for seed generation ... git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6354 6c8d7289-2bf4-0310-a012-ef5d649a1542	2009-09-29 08:05:36 +00:00
orbiter	432154f725	new strategy for concurrent database index key retrieval ... git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6353 6c8d7289-2bf4-0310-a012-ef5d649a1542	2009-09-29 08:04:00 +00:00
orbiter	a11cd9f80f	- removed reverse name lookup for http access logging (grr..) ... - removed a synchronization in seed info string generation git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6351 6c8d7289-2bf4-0310-a012-ef5d649a1542	2009-09-28 15:23:15 +00:00
orbiter	2e6bdce086	- added more logging to balancer ... - changed balancer logic slightly git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6350 6c8d7289-2bf4-0310-a012-ef5d649a1542	2009-09-27 22:35:22 +00:00
orbiter	1171a72006	fix for deadlock as seen in http://forum.yacy-websuche.de/viewtopic.php?p=17521#p17521 ... git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6343 6c8d7289-2bf4-0310-a012-ef5d649a1542	2009-09-24 19:14:35 +00:00
orbiter	031e6eefbd	some updates to dublin core, metadata browsing, file indexing and parser stability ... git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6342 6c8d7289-2bf4-0310-a012-ef5d649a1542	2009-09-24 12:54:45 +00:00
hermens	62a7341c4d	Fix for http://forum.yacy-websuche.de/viewtopic.php?f=5&t=2204 ... git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6341 6c8d7289-2bf4-0310-a012-ef5d649a1542	2009-09-24 11:38:15 +00:00
low012	f65bfaa9af	*) Removed base tag from errror page. This has been added by myself a long time ago as a workaround for some weird behavior of my router, but as it turns out, it does more bad than good in general: If HTTPS is used for communication with YaCy, entering a wrong passwort led to an errror page with a form which would send username and password unencrypted with the user possibly being unaware of this. ... *) changed some comments, added some annotations, added SVN properties here and there git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6340 6c8d7289-2bf4-0310-a012-ef5d649a1542	2009-09-23 21:26:14 +00:00
orbiter	e4797ebcde	fix for http://forum.yacy-websuche.de/viewtopic.php?p=17509#p17509 ... corrupted files are ignored git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6339 6c8d7289-2bf4-0310-a012-ef5d649a1542	2009-09-22 22:00:45 +00:00
orbiter	efa7fb34f0	better oom-awareness of miss-cache in cache ... git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6338 6c8d7289-2bf4-0310-a012-ef5d649a1542	2009-09-22 15:01:29 +00:00
orbiter	3e9dcfc204	fix for http://forum.yacy-websuche.de/viewtopic.php?p=17504#p17504 ... git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6337 6c8d7289-2bf4-0310-a012-ef5d649a1542	2009-09-22 14:39:06 +00:00
orbiter	c3a4aee255	some redesign with a possible fix for the ReferenceContainerCache. ... git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6336 6c8d7289-2bf4-0310-a012-ef5d649a1542	2009-09-22 14:33:57 +00:00
orbiter	aca8a78eb8	fix for shutdown of DocumentIndex objects ... git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6333 6c8d7289-2bf4-0310-a012-ef5d649a1542	2009-09-21 15:24:47 +00:00
orbiter	23ab6fbca4	- navigation appear at correct position when opengeodb-results are also presented after a search ... - show an about box if about.headline and about.body is set git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6332 6c8d7289-2bf4-0310-a012-ef5d649a1542	2009-09-20 22:10:45 +00:00
orbiter	4db34eea73	fix for OOM problem in kelondro Cache ... git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6331 6c8d7289-2bf4-0310-a012-ef5d649a1542	2009-09-19 07:50:46 +00:00
orbiter	8ea1d7ab59	fix for wrong assert condition in search abstract generation ... git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6330 6c8d7289-2bf4-0310-a012-ef5d649a1542	2009-09-18 21:49:38 +00:00
orbiter	fbd77bd77c	git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6328 6c8d7289-2bf4-0310-a012-ef5d649a1542	2009-09-18 12:33:35 +00:00
orbiter	54c7cbf1d9	- fast result for local search in case that less than 10 hits exists ... - small change in display of RAM in profiling git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6326 6c8d7289-2bf4-0310-a012-ef5d649a1542	2009-09-18 09:19:52 +00:00
orbiter	28d4b921b6	different approach for file search ... git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6325 6c8d7289-2bf4-0310-a012-ef5d649a1542	2009-09-18 07:54:16 +00:00
orbiter	f99f86c5c5	added concurrency to file indexing class ... git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6324 6c8d7289-2bf4-0310-a012-ef5d649a1542	2009-09-17 22:37:21 +00:00
orbiter	902d16cf6c	fixes to parser ... git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6323 6c8d7289-2bf4-0310-a012-ef5d649a1542	2009-09-17 22:00:19 +00:00
orbiter	4a1c852435	fix in usage of RAM copy for Table objects and some cosmetics in asserts. ... This bug affected Tables in case that a removeOne() was called and a RAM copy of the table was active. It may happen for peer owners with a lot of RAM assigned to YaCy. The bug appeared especially during crawling when the balancer tried to get new entries from the crawl queue. This bug may help to solve report at http://forum.yacy-websuche.de/viewtopic.php?p=17417#p17417 and will be tracked there git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6322 6c8d7289-2bf4-0310-a012-ef5d649a1542	2009-09-17 20:41:50 +00:00
lotus	dce450e2e0	possible fix for "hung" doc-documents ... git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6320 6c8d7289-2bf4-0310-a012-ef5d649a1542	2009-09-17 17:08:32 +00:00
orbiter	e627f75415	one more fix to badwords and stopwords ... git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6316 6c8d7289-2bf4-0310-a012-ef5d649a1542	2009-09-15 11:47:50 +00:00
orbiter	721b88efbd	- fixed a problem loading blacklists with new yacycore.jar ... - fixed badwords and stopwords initialization git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6315 6c8d7289-2bf4-0310-a012-ef5d649a1542	2009-09-15 11:46:02 +00:00
orbiter	80d5005044	fixed seed upload methods - replaced reflection with direct instantiation ... git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6314 6c8d7289-2bf4-0310-a012-ef5d649a1542	2009-09-15 09:47:02 +00:00
orbiter	68465c37af	added a convenience class to add files into a YaCy index ... to make this possible, the yacyURL must be able to process file:// urls, which has also been implemented testing of the new class resulted in some bugfixes in other classes git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6313 6c8d7289-2bf4-0310-a012-ef5d649a1542	2009-09-14 21:17:42 +00:00
orbiter	2e41e10ffd	- updates to yacyVersion parser (remove old targets) ... - added javadoc target to built script (does not work yet without errors) git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6312 6c8d7289-2bf4-0310-a012-ef5d649a1542	2009-09-14 18:08:40 +00:00
orbiter	27d00285aa	- added a new file reader cache that may serve as full-file-copy of blob database files. This is not yet used ... - removed class FileWriter and replaced all usage of that class with CachedFileWriter git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6309 6c8d7289-2bf4-0310-a012-ef5d649a1542	2009-09-09 21:28:23 +00:00
orbiter	fd6b9cb7dc	refactoring of IO access classes ... git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6308 6c8d7289-2bf4-0310-a012-ef5d649a1542	2009-09-09 20:49:16 +00:00
orbiter	d64569aa39	reuturn only recommendations of words that have a greater count than the original word ... git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6307 6c8d7289-2bf4-0310-a012-ef5d649a1542	2009-09-08 13:57:52 +00:00
orbiter	604c37927f	used comparator for did-you-mean that uses index sizes for comparisment, but: ... - limit comparisment to only the first 10 elements that had been sorted before without IO - added a size cache to index computation because the size is computed at least twice in set comparator git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6306 6c8d7289-2bf4-0310-a012-ef5d649a1542	2009-09-08 13:48:17 +00:00
orbiter	a58d9cae7d	- show location name in geolocalization search result ... - added link from location icon to openstreetmap browser with coordinates git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6305 6c8d7289-2bf4-0310-a012-ef5d649a1542	2009-09-08 10:18:03 +00:00
orbiter	573d03c7d7	added configuration to enable ram table copy ... git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6304 6c8d7289-2bf4-0310-a012-ef5d649a1542	2009-09-07 20:30:57 +00:00
orbiter	3be54e1891	fix to rule when to use a ram table copy ... git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6302 6c8d7289-2bf4-0310-a012-ef5d649a1542	2009-09-07 19:22:12 +00:00
orbiter	700218846c	disabled or removed sleep calls ... git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6301 6c8d7289-2bf4-0310-a012-ef5d649a1542	2009-09-07 18:50:44 +00:00
orbiter	342c5d0fd4	fixed city name detection: finds now also substrings of city names ... git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6300 6c8d7289-2bf4-0310-a012-ef5d649a1542	2009-09-06 20:44:02 +00:00
orbiter	18aa0609ca	fix for caching of word hash computation ... git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6299 6c8d7289-2bf4-0310-a012-ef5d649a1542	2009-09-05 22:05:18 +00:00
orbiter	a10a6cce45	patch for http://forum.yacy-websuche.de/viewtopic.php?p=17289#p17289 ... git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6298 6c8d7289-2bf4-0310-a012-ef5d649a1542	2009-09-05 21:52:50 +00:00
low012	53bbdfd19a	*) setting SVN keywords ... *) minor changes git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6297 6c8d7289-2bf4-0310-a012-ef5d649a1542	2009-09-05 20:41:21 +00:00
low012	25f6145934	*) preventing null pointer exception in case empty search word or only one character is enterd or all search words are removed by filters ... git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6296 6c8d7289-2bf4-0310-a012-ef5d649a1542	2009-09-05 20:31:39 +00:00
low012	248f3fd9b5	*) cleaned up code for better readability ... *) added a few copyright notices *) removed redundancy in constructors of ListToken git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6295 6c8d7289-2bf4-0310-a012-ef5d649a1542	2009-09-05 20:04:23 +00:00
orbiter	eaddf2d464	- corrected layout of map preview ... - added caption to maps containing latitude and longitude information - prevented that maps occur on second search page - added location names to did-you-mean - some refactoring of did-you-mean - added equal and compareTo test to Coordinates class to make that work in set - fixed utf-8 support for library files - fixed a bug in images search icon view caption git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6294 6c8d7289-2bf4-0310-a012-ef5d649a1542	2009-09-04 23:33:47 +00:00
hermens	4b83875abd	Small fixes for the heapCacheIterator in ReferenceContainerCache: ... - Start the iteration at startWordHash - When used with rotation, let the iteration stop when the cache is empty git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6293 6c8d7289-2bf4-0310-a012-ef5d649a1542	2009-09-04 23:27:14 +00:00
orbiter	fd668f531b	fixed map layout ... git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6292 6c8d7289-2bf4-0310-a012-ef5d649a1542	2009-09-04 19:20:27 +00:00
orbiter	2740d9dd79	added integration of osm maps for search ... git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6291 6c8d7289-2bf4-0310-a012-ef5d649a1542	2009-09-04 14:32:36 +00:00
orbiter	af3a696fc4	added a fast-fail concept in search processes. The search now has better control if all the remote searches may bring any result. If all processes are finished, then all search tasks fail fast. ... git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6290 6c8d7289-2bf4-0310-a012-ef5d649a1542	2009-09-03 23:09:53 +00:00
orbiter	ce972ff4ef	update to default ranking profile which has now some settings to deny some phpbb3 pages which are redundant in the index when crawling phpbb3. ... git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6288 6c8d7289-2bf4-0310-a012-ef5d649a1542	2009-09-03 20:54:47 +00:00
orbiter	44579fa06d	- fixed a problem loading images through yacy's document loader, ... this denied non-parseable documents which excluded all images - fixed url of osm tile server git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6287 6c8d7289-2bf4-0310-a012-ef5d649a1542	2009-09-03 11:46:08 +00:00
orbiter	67eddaec4b	changed way to integrate dictionary files: ... the must be downloaded manually by the user and placed in DATA/DICTIONARIES/source for each externally imported dictionary file there will be a translator that converts the input file once into a YaCy-internat data format. Files that will be provided together with yacy releases may still be placed in <root>/dictionaries git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6286 6c8d7289-2bf4-0310-a012-ef5d649a1542	2009-09-02 18:42:13 +00:00
orbiter	d656a94f55	fix for bad paths in dictionary processing ... git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6285 6c8d7289-2bf4-0310-a012-ef5d649a1542	2009-09-02 18:24:41 +00:00
orbiter	3b9aaf9e9f	- inserted new library tests inside DidYouMean ... - some redesign of DidYouMean that was necessary to follow a special rule how a library should be used: - the library provides words that start or end with a test word which may be possibly also an empty set of words - all words that the DidYouMean produced with the four production rules are used to generate a set of library-completed words - if this process results in any words from the library, only library-genrated words are taken - if the is no library-generated word at all, take the artifial generated word - all words that result from these rules are tested against the index - the result is ordered using a lightweight comparator that prefers short words - a not-so-much-io test against the index is beeing prepared next - insered the library initialization into the switchboard git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6284 6c8d7289-2bf4-0310-a012-ef5d649a1542	2009-09-02 13:41:56 +00:00
orbiter	8c35ffe34c	fixes to the dymlib ... git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6283 6c8d7289-2bf4-0310-a012-ef5d649a1542	2009-09-01 15:05:28 +00:00
orbiter	bfa273bcc1	added a library provider which holds libraries in static objects, ... which can be used by any other classes to support their functions. libraries are designed in such a way that users can create and insert their own library files, but can also be imported from other sources. As an example the "Korpusbasierte Wortgrundformliste DeReWo des Institut für Deutsche Sprache" from http://www.ids-mannheim.de has been integrated. This dictionary is licensed to be used for all non-profit purposes. In case that YaCy is used for commercial uses, this library must be removed. The new library provilder reads the original source and translates it into a simple word list to be used for the did-you-mean library provider. More libraries may be provided in the future using a download-servlet which puts files from the internet into the <application-root>/dictionaries/ path. git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6282 6c8d7289-2bf4-0310-a012-ef5d649a1542	2009-09-01 13:54:43 +00:00
orbiter	1762a7bcd6	- moved DidYouMean to the data package ... - added a DidYouMeanLibrary class that shall support the did you mean function with additional word lists git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6281 6c8d7289-2bf4-0310-a012-ef5d649a1542	2009-09-01 13:04:35 +00:00
orbiter	bf8ed00e9e	removed debugging code ... git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6280 6c8d7289-2bf4-0310-a012-ef5d649a1542	2009-08-30 11:03:34 +00:00
orbiter	ead48c4b25	fix for preparation of search result pages with offset > 10: ... - less pages are fetched in advance - just-in-time fetch of next required pages - fix for missing hand-over of offset to fetch threads git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6279 6c8d7289-2bf4-0310-a012-ef5d649a1542	2009-08-30 10:28:23 +00:00
orbiter	39a311d608	better care to do not loose the merge/dump thread ... git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6278 6c8d7289-2bf4-0310-a012-ef5d649a1542	2009-08-29 23:35:03 +00:00
orbiter	10d3e856b5	better concurrency, less blocking & performance hacks ... git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6277 6c8d7289-2bf4-0310-a012-ef5d649a1542	2009-08-29 23:34:14 +00:00
orbiter	1a9cfd8718	some performance hacks (CPU only, not IO) ... this will cause better computation speed for single- and multi-core; there are enhancements that will speed up old and slow machines as well as multi-core CPUs. Indexing of surrogates has been speed up from 4000 PPM to over 20000 PPM on a simple dual core office computer. Since the enhancements are mostly in core routines, the hack should also speed up search performance. git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6276 6c8d7289-2bf4-0310-a012-ef5d649a1542	2009-08-28 13:28:11 +00:00
orbiter	92407009b2	cleanup ... git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6275 6c8d7289-2bf4-0310-a012-ef5d649a1542	2009-08-27 23:20:59 +00:00
orbiter	0ba1beaf56	separated rwi constraint evaluation from rwi ranking and added concurrency ... git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6274 6c8d7289-2bf4-0310-a012-ef5d649a1542	2009-08-27 22:54:32 +00:00
orbiter	ce7924d712	better concurrency for rwi entry parsing during search processing ... git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6273 6c8d7289-2bf4-0310-a012-ef5d649a1542	2009-08-27 22:06:52 +00:00
orbiter	b0637600d5	enhanced url constraint computation: better position of constraint check during retrieval process ... git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6272 6c8d7289-2bf4-0310-a012-ef5d649a1542	2009-08-27 20:20:07 +00:00
orbiter	61748285c3	more refactoring of search ... git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6270 6c8d7289-2bf4-0310-a012-ef5d649a1542	2009-08-27 15:19:48 +00:00
orbiter	323a8e733d	removed unused classes ... git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6269 6c8d7289-2bf4-0310-a012-ef5d649a1542	2009-08-27 14:42:05 +00:00
orbiter	72e5407115	refactoring of snippet cache ... git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6268 6c8d7289-2bf4-0310-a012-ef5d649a1542	2009-08-27 14:34:41 +00:00
orbiter	0e471ba33b	- fixed a bug in fast digest computation ... - added a open-on-demand hack to heap files: when a heap file is opened the first time, it is first scanned to get a key index and then it is closed again. This will free up file pointers in cases where a really large number of blob files are opened upon initialization of ArrayStack objects. This should solve also a problem reported in http://forum.yacy-websuche.de/viewtopic.php?p=17191#p17191 git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6267 6c8d7289-2bf4-0310-a012-ef5d649a1542	2009-08-27 11:03:21 +00:00
low012	93b2622503	*) repaired and added IM online status indicators ... *) added some missing SVN properties *) removed unnecessary comment, added missing copyright notice git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6266 6c8d7289-2bf4-0310-a012-ef5d649a1542	2009-08-26 18:34:00 +00:00
orbiter	e7736d9c8d	more refactoring: made all variables in SearchEvent private ... to prepare splitting of the class into two parts: local and remote search git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6265 6c8d7289-2bf4-0310-a012-ef5d649a1542	2009-08-26 15:59:55 +00:00
orbiter	4b92d0b9b7	patch for possible problems with normalization of '/' in urls. This applies in rare cases when '/' appear in post-properties ... git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6264 6c8d7289-2bf4-0310-a012-ef5d649a1542	2009-08-26 15:10:03 +00:00
orbiter	d8ca6e6bf1	more refactoring for search ... git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6263 6c8d7289-2bf4-0310-a012-ef5d649a1542	2009-08-25 21:27:01 +00:00
orbiter	fe4a4e3f6b	added missing class ... git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6261 6c8d7289-2bf4-0310-a012-ef5d649a1542	2009-08-24 21:03:40 +00:00
orbiter	72ac5bd80f	refactoring of search process. ... this is the beginning of some architecture changes that will hopefully bring some more stability, speed and transparency to the search process. git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6260 6c8d7289-2bf4-0310-a012-ef5d649a1542	2009-08-24 15:24:02 +00:00
hermens	c4d0e22a77	Further speed upof concurrent DHT-receive ... git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6259 6c8d7289-2bf4-0310-a012-ef5d649a1542	2009-08-23 22:37:15 +00:00
hermens	2fbc0696bf	Fix for http://forum.yacy-websuche.de/viewtopic.php?f=6&t=2334 ... git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6258 6c8d7289-2bf4-0310-a012-ef5d649a1542	2009-08-21 13:10:59 +00:00
f1ori	d515bc11e2	added ooxmlparser ... git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6256 6c8d7289-2bf4-0310-a012-ef5d649a1542	2009-08-08 15:34:41 +00:00
orbiter	d9744b1b5d	replaced old caching strategy control class with lightweight simplearc ... git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6254 6c8d7289-2bf4-0310-a012-ef5d649a1542	2009-08-07 23:01:33 +00:00
orbiter	8e56c2ace6	fix for fixes from this afternoon ... git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6253 6c8d7289-2bf4-0310-a012-ef5d649a1542	2009-08-07 22:53:49 +00:00
orbiter	cf739edc2e	fix for possible deadlock, see ... http://forum.yacy-websuche.de/viewtopic.php?p=17017#p17017 git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6252 6c8d7289-2bf4-0310-a012-ef5d649a1542	2009-08-07 12:11:22 +00:00
orbiter	6354b5e447	removed possible deadlock, see ... http://forum.yacy-websuche.de/viewtopic.php?p=17017#p17017 git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6251 6c8d7289-2bf4-0310-a012-ef5d649a1542	2009-08-07 12:04:14 +00:00
orbiter	5cc17ccf8a	a better caching with less overhead and more appropriate ... synchronisation use in more than 10 different data objects git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6250 6c8d7289-2bf4-0310-a012-ef5d649a1542	2009-08-07 11:55:32 +00:00
orbiter	92edd24e70	fixed problem with switching of networks ... git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6247 6c8d7289-2bf4-0310-a012-ef5d649a1542	2009-07-30 15:49:23 +00:00
orbiter	0575f12838	fix for deadlock ... git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6246 6c8d7289-2bf4-0310-a012-ef5d649a1542	2009-07-30 09:08:44 +00:00
orbiter	fbfdaf063d	- patch to omit IndexOutOfBoundsException when a b64-encoded key appears not to be well-formed. In that case the key is still accepted but rated higher than other regular keys to create a virtual ordering between well-formed and ill-formed keys ... - check routine at the beginning of the import of table keys that check that all imported keys are well-formed. All records that have a ill-formed key are deleted. This is a hack and is not tested since I don't have bad data here to test with. If the effect is seen in the wild, please report in the forum. git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6245 6c8d7289-2bf4-0310-a012-ef5d649a1542	2009-07-29 19:43:11 +00:00
orbiter	c0e17de2fb	- fixes for some problems with the new crawling/caching strategies ... - speed enhancements for the cache-only cache policy by using special no-delay rules in the balancer - fixed some deadlock- and 100% CPU problems in the balancer git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6243 6c8d7289-2bf4-0310-a012-ef5d649a1542	2009-07-25 21:38:57 +00:00
orbiter	634a01a9a4	replaced wget-requests with caching requests ... git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6242 6c8d7289-2bf4-0310-a012-ef5d649a1542	2009-07-24 14:52:27 +00:00