orbiter
6192205533
more final modifier
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6407 6c8d7289-2bf4-0310-a012-ef5d649a1542
2009-10-12 21:59:39 +00:00
orbiter
0f6b011e1a
fix for new index location and better way to use own classes by reflection
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6406 6c8d7289-2bf4-0310-a012-ef5d649a1542
2009-10-12 21:12:42 +00:00
orbiter
7a3bbd950f
:-(
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6405 6c8d7289-2bf4-0310-a012-ef5d649a1542
2009-10-12 20:29:03 +00:00
orbiter
b953f04f90
one more reflection fix
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6404 6c8d7289-2bf4-0310-a012-ef5d649a1542
2009-10-12 17:45:42 +00:00
orbiter
77d6604856
fix for npe, see http://forum.yacy-websuche.de/viewtopic.php?p=17727#p17727
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6403 6c8d7289-2bf4-0310-a012-ef5d649a1542
2009-10-12 17:41:16 +00:00
orbiter
2a7fe35f92
performance tuning using more final modifiers in the kelondro core
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6402 6c8d7289-2bf4-0310-a012-ef5d649a1542
2009-10-12 17:37:12 +00:00
orbiter
cb4de9ceee
fixed a bug in table iterator (did not recognize elements in write buffer)
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6401 6c8d7289-2bf4-0310-a012-ef5d649a1542
2009-10-12 08:06:35 +00:00
orbiter
5841ee83d3
refactoring
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6400 6c8d7289-2bf4-0310-a012-ef5d649a1542
2009-10-11 21:29:18 +00:00
orbiter
e7f18ba24b
refactoring
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6399 6c8d7289-2bf4-0310-a012-ef5d649a1542
2009-10-11 00:24:42 +00:00
orbiter
ce8dc575ca
refactoring
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6398 6c8d7289-2bf4-0310-a012-ef5d649a1542
2009-10-11 00:12:19 +00:00
orbiter
bea3b99aff
moved table and util classes
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6397 6c8d7289-2bf4-0310-a012-ef5d649a1542
2009-10-10 01:14:19 +00:00
orbiter
bd876eb4b7
moved io classes
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6396 6c8d7289-2bf4-0310-a012-ef5d649a1542
2009-10-10 01:00:49 +00:00
orbiter
c0e0e1f422
moved blob classes
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6395 6c8d7289-2bf4-0310-a012-ef5d649a1542
2009-10-10 00:43:25 +00:00
orbiter
1e4f8b56ed
accumulated classes from different packages into the new rwi package
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6394 6c8d7289-2bf4-0310-a012-ef5d649a1542
2009-10-10 00:39:15 +00:00
orbiter
194da25a2f
moved kelondro index
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6393 6c8d7289-2bf4-0310-a012-ef5d649a1542
2009-10-09 23:32:08 +00:00
orbiter
4446acc8cd
moved kelondro order
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6392 6c8d7289-2bf4-0310-a012-ef5d649a1542
2009-10-09 23:22:22 +00:00
orbiter
f677d534b1
start of a really extensive refactoring which will produce a hierarchical package structure with the domain yacy.net as package root
...
- moved here the logging classes as part of the new net.yacy.kelondro package
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6391 6c8d7289-2bf4-0310-a012-ef5d649a1542
2009-10-09 23:13:30 +00:00
orbiter
ea473e32b8
refactoring
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6390 6c8d7289-2bf4-0310-a012-ef5d649a1542
2009-10-09 22:27:50 +00:00
orbiter
735e2737e3
* added index segments
...
This is a major change in the organization of indexes.
Please consider a back-up of your data before you run this update.
All existing index files will be moved and renamed to a new position.
With this change, it will be possible to maintain different indexes for different purposes and it will be possible to have a distinction between DHT-in and DHT-out specific indexes. Tenants may also have their own index, and it may be possible to have histories and back-ups of indexes. This is just the beginning, many servlets must be adopted after this change, but all functions that had been there should still work.
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6389 6c8d7289-2bf4-0310-a012-ef5d649a1542
2009-10-09 14:44:20 +00:00
orbiter
09de5da74a
once again a performance hack
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6388 6c8d7289-2bf4-0310-a012-ef5d649a1542
2009-10-08 18:26:54 +00:00
orbiter
2f6d88403e
uä
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6387 6c8d7289-2bf4-0310-a012-ef5d649a1542
2009-10-08 18:10:56 +00:00
orbiter
d2615ea5a8
increased memory for scraper buffer to enhance parsing speed
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6386 6c8d7289-2bf4-0310-a012-ef5d649a1542
2009-10-08 15:27:13 +00:00
orbiter
4bbbb74ec4
removed not necessary synchronization
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6385 6c8d7289-2bf4-0310-a012-ef5d649a1542
2009-10-08 15:26:28 +00:00
hermens
67e5464cc2
Fix for SVN6380: x[] Arrays are unsuitable Keys for Maps without using a proper Comparator.
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6384 6c8d7289-2bf4-0310-a012-ef5d649a1542
2009-10-08 12:55:36 +00:00
lotus
5f72d2b19f
update to jre6u16
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6383 6c8d7289-2bf4-0310-a012-ef5d649a1542
2009-10-08 10:48:33 +00:00
hermens
aeab8c7917
Prevent failed DHT attemps from overwriting newer peer info
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6382 6c8d7289-2bf4-0310-a012-ef5d649a1542
2009-10-08 00:17:29 +00:00
hermens
9324b5b6c5
Enhancements to DHT
...
- speed up deletion of containers when selscted from whole index
- correctly eliminate all references to unavailable URLs, not just the first encountered
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6381 6c8d7289-2bf4-0310-a012-ef5d649a1542
2009-10-08 00:03:16 +00:00
hermens
e49e2d75fe
Limit the time Transmission.Chunks stay in the transmissionCloud by using a Map that iterates entires in insertion order.
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6380 6c8d7289-2bf4-0310-a012-ef5d649a1542
2009-10-07 23:41:25 +00:00
orbiter
92db7c5d07
increased timeout for index retrieval
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6379 6c8d7289-2bf4-0310-a012-ef5d649a1542
2009-10-07 13:03:13 +00:00
lotus
386b9f35f6
activated resource observer for windows 7
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6378 6c8d7289-2bf4-0310-a012-ef5d649a1542
2009-10-07 06:20:24 +00:00
orbiter
6e0dc39a7d
- some fixes to prevent blocking situations
...
- better logging for the crawler
- better default values for the crawler
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6377 6c8d7289-2bf4-0310-a012-ef5d649a1542
2009-10-06 21:52:55 +00:00
orbiter
51f2bbf04b
possible fix for problem in http://forum.yacy-websuche.de/viewtopic.php?p=17655#p17655
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6376 6c8d7289-2bf4-0310-a012-ef5d649a1542
2009-10-06 09:56:14 +00:00
orbiter
f8371707e5
- possibly better termination for SplitTable
...
- better abstraction in DidYouMean
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6375 6c8d7289-2bf4-0310-a012-ef5d649a1542
2009-10-05 22:09:58 +00:00
orbiter
87780f2562
produce did-you-mean also for queries with more than one word
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6374 6c8d7289-2bf4-0310-a012-ef5d649a1542
2009-10-05 21:51:02 +00:00
orbiter
04a548a1e3
- temporary integrated the transferURL servlet as static class instead as a class that is called using reflection to investigate the OOM problems in that class
...
- fixes for numerous other problems
- removed dead code
- resdesign of the strings-method, which produces now less memory overhead and may help to prevent OOMs
- another fix for the deadlock problem in SplitTable
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6373 6c8d7289-2bf4-0310-a012-ef5d649a1542
2009-10-05 20:11:41 +00:00
orbiter
ea427df944
fixed a worst case situation of the condenser which may cause a temporary full CPU load because of a bad data structure usage
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6372 6c8d7289-2bf4-0310-a012-ef5d649a1542
2009-10-05 08:26:55 +00:00
lotus
f1bde59c50
logger config cleanup
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6371 6c8d7289-2bf4-0310-a012-ef5d649a1542
2009-10-02 18:03:14 +00:00
orbiter
3e38035389
fix for interrupted thread during has() property check
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6370 6c8d7289-2bf4-0310-a012-ef5d649a1542
2009-10-02 10:55:40 +00:00
orbiter
5bd1c1d205
just added some comments that had been produced to learn about OAI-PMH
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6369 6c8d7289-2bf4-0310-a012-ef5d649a1542
2009-10-01 22:56:22 +00:00
suessthomas
d52cf19835
small changes to de.lng (parser settings)
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6368 6c8d7289-2bf4-0310-a012-ef5d649a1542
2009-10-01 20:27:05 +00:00
orbiter
6aa474f529
- better logging for web cache access and fail reasons
...
- better Exception handling for web cache access
- distinction between access of web cache for proxy and crawler
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6367 6c8d7289-2bf4-0310-a012-ef5d649a1542
2009-10-01 13:08:19 +00:00
orbiter
3671c37989
added experimental oai-pmh reader and integrated it with the existing dublin core parser
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6366 6c8d7289-2bf4-0310-a012-ef5d649a1542
2009-09-30 22:11:00 +00:00
orbiter
0c17b600c6
remote search by default off
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6365 6c8d7289-2bf4-0310-a012-ef5d649a1542
2009-09-30 15:06:29 +00:00
orbiter
58a00205d5
re-activated the emergency close when too many server connections exist
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6364 6c8d7289-2bf4-0310-a012-ef5d649a1542
2009-09-30 14:29:43 +00:00
orbiter
c57d2070e6
more logging
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6363 6c8d7289-2bf4-0310-a012-ef5d649a1542
2009-09-30 13:25:08 +00:00
orbiter
a995b95367
tried a fix for the httpd access bug (too many unclosed sessions)
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6362 6c8d7289-2bf4-0310-a012-ef5d649a1542
2009-09-30 13:18:02 +00:00
orbiter
e1fba41cad
better logging
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6361 6c8d7289-2bf4-0310-a012-ef5d649a1542
2009-09-29 21:52:17 +00:00
orbiter
2275f885a8
possible fix for concurrency problem
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6360 6c8d7289-2bf4-0310-a012-ef5d649a1542
2009-09-29 21:40:50 +00:00
low012
a6a3090c3d
*) blacklist cleaner supports usage of regular expressions now
...
*) refacored BlacklistCleaner_p.java for better readability
*) moved check of validity of patterns to the Balcklist implementation since patterns might be valid in one implementation, but not in another
*) added method to check validity to Blacklist interface
*) fixed some minor issues like typos or wrong whitespaces
*) set subversion properties for a whole bunch of files
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6359 6c8d7289-2bf4-0310-a012-ef5d649a1542
2009-09-29 21:28:49 +00:00
orbiter
5a93807781
improved web cache speed:
...
- removed one computation out of a synchronization
- removed one not necessary has() call
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6358 6c8d7289-2bf4-0310-a012-ef5d649a1542
2009-09-29 08:41:05 +00:00