Marown/yacy_search_server
1
0
Fork 0
mirror of https://github.com/yacy/yacy_search_server.git synced 2024-09-19 00:01:41 +02:00
Code Issues Actions Packages Projects Releases Wiki Activity
6,449 Commits 1 Branch 20 Tags 836 MiB
a85c5bb8a7
Commit Graph

4275 Commits

Author SHA1 Message Date
orbiter
4c99d4683d possible fix for lost crawl profile handles: clean-up job did wrong measurement to see if crawl is still running. 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6465 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-11-06 23:15:20 +00:00
orbiter
cd6745b292 accept rss feeds without channel descriptions 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6464 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-11-06 22:46:21 +00:00
orbiter
08f1cbb125 another update to the pdf parser 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6463 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-11-06 22:41:37 +00:00
orbiter
54c54fb144 get a handle for grep: 'StackTrace' 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6462 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-11-06 19:55:21 +00:00
orbiter
605e896d6c more details for exception catching when parsing pdfs 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6461 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-11-06 19:47:24 +00:00
orbiter
18b21eaffe small fixes to search default values and server logging 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6460 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-11-06 19:13:35 +00:00
lotus
6edc168cfe option to disable dht by memory limit: 
memory.acceptDHT in kbytes
not yet pre-enabled, will clear on every startup
please review since this could break dht in freeworld

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6459 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-11-06 19:13:30 +00:00
orbiter
4431b9767e added about 450 replacements for printStackTrace() methods to pipe such traces into the log at DATA/LOG/ 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6458 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-11-05 20:28:37 +00:00
orbiter
e3025ee691 - new icon for OAI-PMH loading action 
- added many stack trace outputs for exceptions in crawl profile handler to find the 'missing profile handle' bug
- catched one more timeout exception in httpd file loader

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6457 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-11-05 16:40:15 +00:00
orbiter
f0b8db93f0 - more abstraction of serverCore thread access 
- no more keep-alive when number of connections exceeds 1/2 of the allowed number of connection

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6456 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-11-05 14:54:43 +00:00
orbiter
19f31bb043 - moved OAI-PMH source list file from SETTINGS to DICTIONARIES/harvesting 
- added convenience method for loading of files from the web in LoaderDispatcher

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6455 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-11-04 16:15:28 +00:00
orbiter
2889b9426e missing code for last commit 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6454 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-11-04 12:03:19 +00:00
orbiter
b6a8887ff5 better handling of running sessions without explicit hashtable 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6453 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-11-04 11:59:15 +00:00
orbiter
1dc7ea986a added a dynamic keep-alive time-out for http server sessions: 
if there are many concurrent server sessions, the timout is decreased.
This should avoid a situation where the clean-up thread is too
late to stop running http sessions that should be terminated
before the maximum number of server sessions is reached.

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6452 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-11-04 11:01:09 +00:00
low012
e77c906673 *) minor changes mainly in comments 
*) added svn:keyword settings for several files

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6451 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-11-03 22:47:53 +00:00
low012
f1740edbf8 *) added skript to change memory settings, password and port (experimental, don't blame me if it messes up your configuration) 
*) minor change in Digest class, added option in main method, might not be optimal yet

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6450 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-11-03 22:28:29 +00:00
orbiter
11f7da06ed - fixes to csv parser 
- automatic OAI-PMH import by just clicking on one link from the provided resource list

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6449 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-11-03 21:18:19 +00:00
orbiter
9b6762ec2e - added a csv "comma separated values" parser to parse OAI-PMH sources from 
http://roar.eprints.org/index.php?action=csv
- integrated the csv parser into the crawlers parser list
- added an extension to the OAI-PMH import function to download and show the roar csv file using the csv parser

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6448 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-11-03 20:10:59 +00:00
orbiter
176e334aa4 fixes 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6446 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-11-02 19:23:05 +00:00
orbiter
2fa6bf440b workflow update to OAI-PMH importer 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6445 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-11-02 18:19:30 +00:00
orbiter
b0b7a4f9a5 - added function to OAI-PMH reader that can pull all records from a server using an evaluation of the resumption token to get URL to retrieve remaining records 
- added monitoring for retrieved records

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6444 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-11-02 11:53:14 +00:00
orbiter
350d13e153 very first working version of oai-pmh importer: if given the right url, the importer can read and index listRecord xml files and calculate the right resumptionURL which is then given as next default start point for the importer url input. 
no automatic harvesting by now, this will be done later

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6443 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-11-02 00:14:14 +00:00
lotus
58616d99e4 patch for yacy disk usage detection on lvm host 
by Michael S.

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6442 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-11-01 08:54:16 +00:00
lotus
79251e6f60 configurable disk space hardlimit for dht 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6441 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-10-31 19:12:53 +00:00
orbiter
a0e891c63d - some redesign in UI menu structure to make room for new 'Content Integration' main menu containing import servlets for Wikimedia Dumps, phpbb3 forum imports and OAI-PMH imports 
- extended the OAI-PMH test applet and integrated it into the menu. Does still not import OAI-PMH records, but shows that it is able to read and parse this data
- some redesign in ZURL storage: refactoring of access methods, better concurrency, less synchronization
- added a limitation to the LURL metadata database table cache to 20 million entries: this cache was until now not limited and only limited by the available RAM which may have caused a memory-leak-like behavior.

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6440 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-10-31 11:58:06 +00:00
orbiter
4240785f20 added anti-alias function for line drawing 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6438 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-10-30 15:58:36 +00:00
orbiter
30f108f97d added stub of oai-pmh importer (not working yet) 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6437 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-10-30 15:58:04 +00:00
orbiter
77c99e500f added more control over memory allocation 
should avoid some of the OOMs

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6436 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-10-27 15:25:48 +00:00
orbiter
52470d0de4 - fix for xls parser 
- fix for image parser
- temporary integration of images as document types in the crawler and indexer for testing of the image parser

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6435 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-10-22 22:38:04 +00:00
orbiter
5e8038ac4d - refactoring of blacklists 
- refactoring of event origin encoding


git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6434 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-10-21 20:14:30 +00:00
orbiter
26fafd85a5 - more refactoring 
- fixed problem with parsers

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6433 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-10-21 15:12:34 +00:00
orbiter
3528b970d6 - refactoring 
- added new experimental (not-yet-working) image parser
- added new test image

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6431 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-10-19 22:34:44 +00:00
orbiter
a8ce192f63 - shifted main classes to new package net.yacy 
- fixed some bugs in last commit

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6427 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-10-18 01:38:07 +00:00
orbiter
b79f4f062f refactoring of yacy documents and parsers: they depend now only on the kelondro classes 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6426 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-10-18 00:53:43 +00:00
hermens
0fd9540866 Configuration of HTTPDProxyHandler logging 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6425 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-10-17 14:04:18 +00:00
orbiter
cee7a05ff2 - de-serialized the pdf parser 
- added fail callback for file indexer

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6415 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-10-15 10:47:29 +00:00
orbiter
9db928ce53 replaced fontbox 0.7.3 with fontbox 0.8.0 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6414 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-10-15 09:51:16 +00:00
orbiter
c2272785c7 - fix for xlsx and pptx parsing 
- less exception logging for swf parser

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6413 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-10-14 19:15:38 +00:00
orbiter
c864901087 - moved httpd.mime to defaults path 
- some documentation fixes
- adopted a default setting for the search window: moves css setting to base.css
- some enhancements for the DocumentIndex class

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6410 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-10-14 13:29:09 +00:00
low012
8829ec5f18 *) made sure that   is replaced with a space and not just deleted in CharacterCoding.java 
*) added annotations and made minor changes to serverObjects.java
*) set subversion properties for several files

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6409 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-10-13 20:57:56 +00:00
orbiter
6c347a37eb more options for DocumentIndex 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6408 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-10-13 08:43:02 +00:00
orbiter
6192205533 more final modifier 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6407 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-10-12 21:59:39 +00:00
orbiter
0f6b011e1a fix for new index location and better way to use own classes by reflection 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6406 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-10-12 21:12:42 +00:00
orbiter
7a3bbd950f :-( 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6405 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-10-12 20:29:03 +00:00
orbiter
b953f04f90 one more reflection fix 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6404 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-10-12 17:45:42 +00:00
orbiter
77d6604856 fix for npe, see http://forum.yacy-websuche.de/viewtopic.php?p=17727#p17727 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6403 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-10-12 17:41:16 +00:00
orbiter
2a7fe35f92 performance tuning using more final modifiers in the kelondro core 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6402 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-10-12 17:37:12 +00:00
orbiter
cb4de9ceee fixed a bug in table iterator (did not recognize elements in write buffer) 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6401 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-10-12 08:06:35 +00:00
orbiter
e7f18ba24b refactoring 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6399 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-10-11 00:24:42 +00:00
orbiter
ce8dc575ca refactoring 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6398 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-10-11 00:12:19 +00:00
orbiter
bea3b99aff moved table and util classes 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6397 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-10-10 01:14:19 +00:00
orbiter
bd876eb4b7 moved io classes 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6396 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-10-10 01:00:49 +00:00
orbiter
c0e0e1f422 moved blob classes 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6395 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-10-10 00:43:25 +00:00
orbiter
1e4f8b56ed accumulated classes from different packages into the new rwi package 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6394 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-10-10 00:39:15 +00:00
orbiter
194da25a2f moved kelondro index 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6393 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-10-09 23:32:08 +00:00
orbiter
4446acc8cd moved kelondro order 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6392 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-10-09 23:22:22 +00:00
orbiter
f677d534b1 start of a really extensive refactoring which will produce a hierarchical package structure with the domain yacy.net as package root 
- moved here the logging classes as part of the new net.yacy.kelondro package

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6391 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-10-09 23:13:30 +00:00
orbiter
ea473e32b8 refactoring 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6390 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-10-09 22:27:50 +00:00
orbiter
735e2737e3 * added index segments 
This is a major change in the organization of indexes.
Please consider a back-up of your data before you run this update.
All existing index files will be moved and renamed to a new position.
With this change, it will be possible to maintain different indexes for different purposes and it will be possible to have a distinction between DHT-in and DHT-out specific indexes. Tenants may also have their own index, and it may be possible to have histories and back-ups of indexes. This is just the beginning, many servlets must be adopted after this change, but all functions that had been there should still work.

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6389 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-10-09 14:44:20 +00:00
orbiter
09de5da74a once again a performance hack 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6388 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-10-08 18:26:54 +00:00
orbiter
2f6d88403e uä 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6387 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-10-08 18:10:56 +00:00
orbiter
d2615ea5a8 increased memory for scraper buffer to enhance parsing speed 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6386 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-10-08 15:27:13 +00:00
orbiter
4bbbb74ec4 removed not necessary synchronization 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6385 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-10-08 15:26:28 +00:00
hermens
67e5464cc2 Fix for SVN6380: x[] Arrays are unsuitable Keys for Maps without using a proper Comparator. 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6384 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-10-08 12:55:36 +00:00
hermens
aeab8c7917 Prevent failed DHT attemps from overwriting newer peer info 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6382 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-10-08 00:17:29 +00:00
hermens
9324b5b6c5 Enhancements to DHT 
- speed up deletion of containers when selscted from whole index
- correctly eliminate all references to unavailable URLs, not just the first encountered



git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6381 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-10-08 00:03:16 +00:00
hermens
e49e2d75fe Limit the time Transmission.Chunks stay in the transmissionCloud by using a Map that iterates entires in insertion order. 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6380 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-10-07 23:41:25 +00:00
orbiter
92db7c5d07 increased timeout for index retrieval 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6379 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-10-07 13:03:13 +00:00
lotus
386b9f35f6 activated resource observer for windows 7 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6378 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-10-07 06:20:24 +00:00
orbiter
6e0dc39a7d - some fixes to prevent blocking situations 
- better logging for the crawler
- better default values for the crawler

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6377 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-10-06 21:52:55 +00:00
orbiter
51f2bbf04b possible fix for problem in http://forum.yacy-websuche.de/viewtopic.php?p=17655#p17655 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6376 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-10-06 09:56:14 +00:00
orbiter
f8371707e5 - possibly better termination for SplitTable 
- better abstraction in DidYouMean

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6375 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-10-05 22:09:58 +00:00
orbiter
87780f2562 produce did-you-mean also for queries with more than one word 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6374 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-10-05 21:51:02 +00:00
orbiter
04a548a1e3 - temporary integrated the transferURL servlet as static class instead as a class that is called using reflection to investigate the OOM problems in that class 
- fixes for numerous other problems
- removed dead code
- resdesign of the strings-method, which produces now less memory overhead and may help to prevent OOMs
- another fix for the deadlock problem in SplitTable

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6373 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-10-05 20:11:41 +00:00
orbiter
ea427df944 fixed a worst case situation of the condenser which may cause a temporary full CPU load because of a bad data structure usage 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6372 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-10-05 08:26:55 +00:00
orbiter
3e38035389 fix for interrupted thread during has() property check 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6370 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-10-02 10:55:40 +00:00
orbiter
5bd1c1d205 just added some comments that had been produced to learn about OAI-PMH 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6369 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-10-01 22:56:22 +00:00
orbiter
6aa474f529 - better logging for web cache access and fail reasons 
- better Exception handling for web cache access
- distinction between access of web cache for proxy and crawler


git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6367 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-10-01 13:08:19 +00:00
orbiter
3671c37989 added experimental oai-pmh reader and integrated it with the existing dublin core parser 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6366 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-09-30 22:11:00 +00:00
orbiter
58a00205d5 re-activated the emergency close when too many server connections exist 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6364 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-09-30 14:29:43 +00:00
orbiter
c57d2070e6 more logging 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6363 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-09-30 13:25:08 +00:00
orbiter
a995b95367 tried a fix for the httpd access bug (too many unclosed sessions) 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6362 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-09-30 13:18:02 +00:00
orbiter
e1fba41cad better logging 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6361 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-09-29 21:52:17 +00:00
orbiter
2275f885a8 possible fix for concurrency problem 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6360 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-09-29 21:40:50 +00:00
low012
a6a3090c3d *) blacklist cleaner supports usage of regular expressions now 
*) refacored BlacklistCleaner_p.java for better readability
*) moved check of validity of patterns to the Balcklist implementation since patterns might be valid in one implementation, but not in another
*) added method to check validity to Blacklist interface
*) fixed some minor issues like typos or wrong whitespaces
*) set subversion properties for a whole bunch of files

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6359 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-09-29 21:28:49 +00:00
orbiter
5a93807781 improved web cache speed: 
- removed one computation out of a synchronization
- removed one not necessary has() call


git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6358 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-09-29 08:41:05 +00:00
orbiter
2e8b2867ff double performance of store method because it avoids one 'has' 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6357 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-09-29 08:23:44 +00:00
orbiter
afda5b1adc new join method for indexes (not yet used) 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6356 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-09-29 08:16:24 +00:00
orbiter
65b66c2c18 better handling of array files of length 0 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6355 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-09-29 08:13:44 +00:00
orbiter
1957b5797a fix for seed generation 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6354 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-09-29 08:05:36 +00:00
orbiter
432154f725 new strategy for concurrent database index key retrieval 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6353 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-09-29 08:04:00 +00:00
orbiter
a11cd9f80f - removed reverse name lookup for http access logging (grr..) 
- removed a synchronization in seed info string generation

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6351 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-09-28 15:23:15 +00:00
orbiter
2e6bdce086 - added more logging to balancer 
- changed balancer logic slightly

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6350 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-09-27 22:35:22 +00:00
orbiter
1171a72006 fix for deadlock as seen in http://forum.yacy-websuche.de/viewtopic.php?p=17521#p17521 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6343 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-09-24 19:14:35 +00:00
orbiter
031e6eefbd some updates to dublin core, metadata browsing, file indexing and parser stability 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6342 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-09-24 12:54:45 +00:00
hermens
62a7341c4d Fix for http://forum.yacy-websuche.de/viewtopic.php?f=5&t=2204 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6341 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-09-24 11:38:15 +00:00
low012
f65bfaa9af *) Removed base tag from errror page. This has been added by myself a long time ago as a workaround for some weird behavior of my router, but as it turns out, it does more bad than good in general: If HTTPS is used for communication with YaCy, entering a wrong passwort led to an errror page with a form which would send username and password unencrypted with the user possibly being unaware of this. 
*) changed some comments, added some annotations, added SVN properties here and there

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6340 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-09-23 21:26:14 +00:00
orbiter
e4797ebcde fix for http://forum.yacy-websuche.de/viewtopic.php?p=17509#p17509 
corrupted files are ignored

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6339 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-09-22 22:00:45 +00:00
orbiter
efa7fb34f0 better oom-awareness of miss-cache in cache 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6338 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-09-22 15:01:29 +00:00
orbiter
3e9dcfc204 fix for http://forum.yacy-websuche.de/viewtopic.php?p=17504#p17504 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6337 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-09-22 14:39:06 +00:00
orbiter
c3a4aee255 some redesign with a possible fix for the ReferenceContainerCache. 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6336 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-09-22 14:33:57 +00:00
orbiter
aca8a78eb8 fix for shutdown of DocumentIndex objects 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6333 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-09-21 15:24:47 +00:00
orbiter
23ab6fbca4 - navigation appear at correct position when opengeodb-results are also presented after a search 
- show an about box if about.headline and about.body is set

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6332 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-09-20 22:10:45 +00:00
orbiter
4db34eea73 fix for OOM problem in kelondro Cache 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6331 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-09-19 07:50:46 +00:00
orbiter
8ea1d7ab59 fix for wrong assert condition in search abstract generation 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6330 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-09-18 21:49:38 +00:00
orbiter
fbd77bd77c git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6328 6c8d7289-2bf4-0310-a012-ef5d649a1542 2009-09-18 12:33:35 +00:00
orbiter
54c7cbf1d9 - fast result for local search in case that less than 10 hits exists 
- small change in display of RAM in profiling

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6326 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-09-18 09:19:52 +00:00
orbiter
28d4b921b6 different approach for file search 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6325 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-09-18 07:54:16 +00:00
orbiter
f99f86c5c5 added concurrency to file indexing class 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6324 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-09-17 22:37:21 +00:00
orbiter
902d16cf6c fixes to parser 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6323 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-09-17 22:00:19 +00:00
orbiter
4a1c852435 fix in usage of RAM copy for Table objects and some cosmetics in asserts. 
This bug affected Tables in case that a removeOne() was called and a RAM copy of the table was active. It may happen for peer owners with a lot of RAM assigned to YaCy. The bug appeared especially during crawling when the balancer tried to get new entries from the crawl queue.
This bug may help to solve report at
http://forum.yacy-websuche.de/viewtopic.php?p=17417#p17417
and will be tracked there

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6322 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-09-17 20:41:50 +00:00
lotus
dce450e2e0 possible fix for "hung" doc-documents 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6320 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-09-17 17:08:32 +00:00
orbiter
e627f75415 one more fix to badwords and stopwords 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6316 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-09-15 11:47:50 +00:00
orbiter
721b88efbd - fixed a problem loading blacklists with new yacycore.jar 
- fixed badwords and stopwords initialization

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6315 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-09-15 11:46:02 +00:00
orbiter
80d5005044 fixed seed upload methods - replaced reflection with direct instantiation 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6314 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-09-15 09:47:02 +00:00
orbiter
68465c37af added a convenience class to add files into a YaCy index 
to make this possible, the yacyURL must be able to process file:// urls, which has also been implemented
testing of the new class resulted in some bugfixes in other classes

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6313 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-09-14 21:17:42 +00:00
orbiter
2e41e10ffd - updates to yacyVersion parser (remove old targets) 
- added javadoc target to built script (does not work yet without errors)

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6312 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-09-14 18:08:40 +00:00
orbiter
27d00285aa - added a new file reader cache that may serve as full-file-copy of blob database files. This is not yet used 
- removed class FileWriter and replaced all usage of that class with CachedFileWriter

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6309 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-09-09 21:28:23 +00:00
orbiter
fd6b9cb7dc refactoring of IO access classes 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6308 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-09-09 20:49:16 +00:00
orbiter
d64569aa39 reuturn only recommendations of words that have a greater count than the original word 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6307 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-09-08 13:57:52 +00:00
orbiter
604c37927f used comparator for did-you-mean that uses index sizes for comparisment, but: 
- limit comparisment to only the first 10 elements that had been sorted before without IO
- added a size cache to index computation because the size is computed at least twice in set comparator


git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6306 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-09-08 13:48:17 +00:00
orbiter
a58d9cae7d - show location name in geolocalization search result 
- added link from location icon to openstreetmap browser with coordinates

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6305 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-09-08 10:18:03 +00:00
orbiter
573d03c7d7 added configuration to enable ram table copy 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6304 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-09-07 20:30:57 +00:00
orbiter
3be54e1891 fix to rule when to use a ram table copy 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6302 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-09-07 19:22:12 +00:00
orbiter
700218846c disabled or removed sleep calls 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6301 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-09-07 18:50:44 +00:00
orbiter
342c5d0fd4 fixed city name detection: finds now also substrings of city names 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6300 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-09-06 20:44:02 +00:00
orbiter
18aa0609ca fix for caching of word hash computation 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6299 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-09-05 22:05:18 +00:00
orbiter
a10a6cce45 patch for http://forum.yacy-websuche.de/viewtopic.php?p=17289#p17289 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6298 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-09-05 21:52:50 +00:00
low012
53bbdfd19a *) setting SVN keywords 
*) minor changes

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6297 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-09-05 20:41:21 +00:00
low012
25f6145934 *) preventing null pointer exception in case empty search word or only one character is enterd or all search words are removed by filters 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6296 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-09-05 20:31:39 +00:00
low012
248f3fd9b5 *) cleaned up code for better readability 
*) added a few copyright notices
*) removed redundancy in constructors of ListToken

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6295 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-09-05 20:04:23 +00:00
orbiter
eaddf2d464 - corrected layout of map preview 
- added caption to maps containing latitude and longitude information
- prevented that maps occur on second search page
- added location names to did-you-mean
- some refactoring of did-you-mean
- added equal and compareTo test to Coordinates class to make that work in set
- fixed utf-8 support for library files
- fixed a bug in images search icon view caption

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6294 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-09-04 23:33:47 +00:00
hermens
4b83875abd Small fixes for the heapCacheIterator in ReferenceContainerCache: 
- Start the iteration at startWordHash
- When used with rotation, let the iteration stop when the cache is empty


git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6293 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-09-04 23:27:14 +00:00
orbiter
fd668f531b fixed map layout 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6292 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-09-04 19:20:27 +00:00
orbiter
2740d9dd79 added integration of osm maps for search 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6291 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-09-04 14:32:36 +00:00
orbiter
af3a696fc4 added a fast-fail concept in search processes. The search now has better control if all the remote searches may bring any result. If all processes are finished, then all search tasks fail fast. 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6290 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-09-03 23:09:53 +00:00
orbiter
ce972ff4ef update to default ranking profile which has now some settings to deny some phpbb3 pages which are redundant in the index when crawling phpbb3. 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6288 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-09-03 20:54:47 +00:00
orbiter
44579fa06d - fixed a problem loading images through yacy's document loader, 
this denied non-parseable documents which excluded all images
- fixed url of osm tile server

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6287 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-09-03 11:46:08 +00:00
orbiter
67eddaec4b changed way to integrate dictionary files: 
the must be downloaded manually by the user and placed in DATA/DICTIONARIES/source
for each externally imported dictionary file there will be a translator that converts the input file once
into a YaCy-internat data format.
Files that will be provided together with yacy releases may still be placed in <root>/dictionaries

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6286 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-09-02 18:42:13 +00:00
orbiter
d656a94f55 fix for bad paths in dictionary processing 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6285 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-09-02 18:24:41 +00:00
orbiter
3b9aaf9e9f - inserted new library tests inside DidYouMean 
- some redesign of DidYouMean that was necessary to follow
  a special rule how a library should be used:
  - the library provides words that start or end with a test
    word which may be possibly also an empty set of words
  - all words that the DidYouMean produced with the four
    production rules are used to generate a set of
    library-completed words
  - if this process results in any words from the library,
    only library-genrated words are taken
  - if the is no library-generated word at all, take the
    artifial generated word
  - all words that result from these rules are tested against
    the index
  - the result is ordered using a lightweight comparator that
    prefers short words
  - a not-so-much-io test against the index is beeing prepared
    next
- insered the library initialization into the switchboard

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6284 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-09-02 13:41:56 +00:00
orbiter
8c35ffe34c fixes to the dymlib 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6283 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-09-01 15:05:28 +00:00
orbiter
bfa273bcc1 added a library provider which holds libraries in static objects, 
which can be used by any other classes to support their functions.
libraries are designed in such a way that users can create and
insert their own library files, but can also be imported from
other sources. As an example the "Korpusbasierte Wortgrundformliste
DeReWo des Institut für Deutsche Sprache" from
http://www.ids-mannheim.de has been integrated. This dictionary
is licensed to be used for all non-profit purposes. In case that
YaCy is used for commercial uses, this library must be removed.
The new library provilder reads the original source and translates
it into a simple word list to be used for the did-you-mean library
provider. More libraries may be provided in the future using
a download-servlet which puts files from the internet into the
<application-root>/dictionaries/ path.

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6282 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-09-01 13:54:43 +00:00
orbiter
1762a7bcd6 - moved DidYouMean to the data package 
- added a DidYouMeanLibrary class that shall support the did you mean function with additional word lists


git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6281 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-09-01 13:04:35 +00:00
orbiter
bf8ed00e9e removed debugging code 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6280 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-08-30 11:03:34 +00:00
orbiter
ead48c4b25 fix for preparation of search result pages with offset > 10: 
- less pages are fetched in advance
- just-in-time fetch of next required pages
- fix for missing hand-over of offset to fetch threads

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6279 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-08-30 10:28:23 +00:00
orbiter
39a311d608 better care to do not loose the merge/dump thread 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6278 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-08-29 23:35:03 +00:00
orbiter
10d3e856b5 better concurrency, less blocking & performance hacks 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6277 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-08-29 23:34:14 +00:00
orbiter
1a9cfd8718 some performance hacks (CPU only, not IO) 
this will cause better computation speed for single- and multi-core;
there are enhancements that will speed up old and slow machines as well
as multi-core CPUs. Indexing of surrogates has been speed up
from 4000 PPM to over 20000 PPM on a simple dual core office computer.
Since the enhancements are mostly in core routines, the hack should also
speed up search performance.

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6276 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-08-28 13:28:11 +00:00
orbiter
92407009b2 cleanup 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6275 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-08-27 23:20:59 +00:00
orbiter
0ba1beaf56 separated rwi constraint evaluation from rwi ranking and added concurrency 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6274 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-08-27 22:54:32 +00:00
orbiter
ce7924d712 better concurrency for rwi entry parsing during search processing 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6273 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-08-27 22:06:52 +00:00
orbiter
b0637600d5 enhanced url constraint computation: better position of constraint check during retrieval process 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6272 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-08-27 20:20:07 +00:00
orbiter
61748285c3 more refactoring of search 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6270 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-08-27 15:19:48 +00:00
orbiter
323a8e733d removed unused classes 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6269 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-08-27 14:42:05 +00:00
orbiter
72e5407115 refactoring of snippet cache 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6268 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-08-27 14:34:41 +00:00
orbiter
0e471ba33b - fixed a bug in fast digest computation 
- added a open-on-demand hack to heap files: when a heap file is
  opened the first time, it is first scanned to get a key index
  and then it is closed again. This will free up file pointers
  in cases where a really large number of blob files are opened
  upon initialization of ArrayStack objects. This should solve
  also a problem reported in
  http://forum.yacy-websuche.de/viewtopic.php?p=17191#p17191

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6267 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-08-27 11:03:21 +00:00
low012
93b2622503 *) repaired and added IM online status indicators 
*) added some missing SVN properties
*) removed unnecessary comment, added missing copyright notice

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6266 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-08-26 18:34:00 +00:00
orbiter
e7736d9c8d more refactoring: made all variables in SearchEvent private 
to prepare splitting of the class into two parts: local and remote search

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6265 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-08-26 15:59:55 +00:00
orbiter
4b92d0b9b7 patch for possible problems with normalization of '/' in urls. This applies in rare cases when '/' appear in post-properties 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6264 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-08-26 15:10:03 +00:00
orbiter
d8ca6e6bf1 more refactoring for search 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6263 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-08-25 21:27:01 +00:00
orbiter
fe4a4e3f6b added missing class 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6261 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-08-24 21:03:40 +00:00
orbiter
72ac5bd80f refactoring of search process. 
this is the beginning of some architecture changes that will hopefully bring some more stability, speed and transparency to the search process.

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6260 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-08-24 15:24:02 +00:00
hermens
c4d0e22a77 Further speed upof concurrent DHT-receive 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6259 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-08-23 22:37:15 +00:00
hermens
2fbc0696bf Fix for http://forum.yacy-websuche.de/viewtopic.php?f=6&t=2334 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6258 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-08-21 13:10:59 +00:00
f1ori
d515bc11e2 added ooxmlparser 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6256 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-08-08 15:34:41 +00:00
orbiter
d9744b1b5d replaced old caching strategy control class with lightweight simplearc 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6254 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-08-07 23:01:33 +00:00
orbiter
8e56c2ace6 fix for fixes from this afternoon 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6253 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-08-07 22:53:49 +00:00
orbiter
cf739edc2e fix for possible deadlock, see 
http://forum.yacy-websuche.de/viewtopic.php?p=17017#p17017

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6252 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-08-07 12:11:22 +00:00
orbiter
6354b5e447 removed possible deadlock, see 
http://forum.yacy-websuche.de/viewtopic.php?p=17017#p17017

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6251 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-08-07 12:04:14 +00:00
orbiter
5cc17ccf8a a better caching with less overhead and more appropriate 
synchronisation use in more than 10 different data objects

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6250 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-08-07 11:55:32 +00:00
orbiter
92edd24e70 fixed problem with switching of networks 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6247 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-07-30 15:49:23 +00:00
orbiter
0575f12838 fix for deadlock 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6246 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-07-30 09:08:44 +00:00
orbiter
fbfdaf063d - patch to omit IndexOutOfBoundsException when a b64-encoded key appears not to be well-formed. In that case the key is still accepted but rated higher than other regular keys to create a virtual ordering between well-formed and ill-formed keys 
- check routine at the beginning of the import of table keys that check that all imported keys are well-formed. All records that have a ill-formed key are deleted. This is a hack and is not tested since I don't have bad data here to test with. If the effect is seen in the wild, please report in the forum.


git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6245 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-07-29 19:43:11 +00:00
orbiter
c0e17de2fb - fixes for some problems with the new crawling/caching strategies 
- speed enhancements for the cache-only cache policy by using special no-delay rules in the balancer
- fixed some deadlock- and 100% CPU problems in the balancer

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6243 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-07-25 21:38:57 +00:00
orbiter
634a01a9a4 replaced wget-requests with caching requests 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6242 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-07-24 14:52:27 +00:00
orbiter
c6c97f23ad - added cache usage properties to crawl start 
- added special rule to balancer to omit forced delays if cache is used exclusively
- extended the htCache size by default to 32GB

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6241 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-07-24 11:54:04 +00:00
orbiter
c4ae2cd03f fixed bug that caused deletion of crawl profiles at every application startup 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6240 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-07-23 22:09:02 +00:00
orbiter
161d2fd2ef redesign of access to the HTCache (now http.client.Cache): 
- better control to the cache by using combined request-header and content access methods
- refactoring of many classes to comply to this new access method
- make shure that the cache is always written if something was loaded
- some redesign of the process how http response results are feeded into the new indexing queue
- introduction of a cache read policy:
 * never use the cache
 * use the cache if entry exist
 * use the cache if the proxy freshness rule confirmes
 * use only the cache and go never online
- added configuration options for the crawl profiles to use the new cache policies. There is not yet a input during crawl start to set the policy but this will be added in another step.
- set the default policies for the existing crawl profiles. If you want them to appear in your default profiles you must delete the crawl profiles database; othervise the policy is 'proxy freshness rule'
- enhanced some cache access methods in such a way that unnecessary retrievals are omitted (i.e. for size computation). That should reduce some IO but also a lot of CPU computation because sizes were computed after decompression of content after retrieval of the content from the disc.

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6239 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-07-23 21:31:51 +00:00
f1ori
ba2e6de538 fix empty version string again 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6236 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-07-21 19:56:40 +00:00
orbiter
51534df0cb fix for possible synchronization problem 
see also: http://forum.yacy-websuche.de/viewtopic.php?f=6&t=2292&hilit=&p=16787#p16787

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6234 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-07-20 08:21:17 +00:00
orbiter
4da9042e8a code simplification 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6233 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-07-19 21:59:29 +00:00
orbiter
1d8d51075c refactoring: 
- removed the plasma package. The name of that package came from a very early pre-version of YaCy, even before YaCy was named AnomicHTTPProxy. The Proxy project introduced search for cache contents using class files that had been developed during the plasma project. Information from 2002 about plasma can be found here:
http://web.archive.org/web/20020802110827/http://anomic.de/AnomicPlasma/index.html
We stil have one class that comes mostly unchanged from the plasma project, the Condenser class. But this is now part of the document package and all other classes in the plasma package can be assigned to other packages.
- cleaned up the http package: better structure of that class and clean isolation of server and client classes. The old HTCache becomes part of the client sub-package of http.
- because the plasmaSwitchboard is now part of the search package all servlets had to be touched to declare a different package source.

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6232 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-07-19 20:37:44 +00:00
f1ori
67da20647f * add new odf parser based on sax-xml-parser 
* remove odf_utils-jar
* test metadata in ParserTest


git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6231 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-07-18 15:04:34 +00:00
f1ori
6d0e6d591b * ops, fix compiler error :( 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6227 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-07-17 20:02:56 +00:00
f1ori
3e5beb1654 * fix for empty version in seedlist 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6226 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-07-17 19:16:26 +00:00
orbiter
5bb8074150 removed the indexing queue. This queue was superfluous since the introduction of the blocking queues last year, where documents are parsed, analysed and stored in the index with concurrency. 
- The indexing queue was a historic data structure that was introduced at the very beginning at the project as a part of the switchboard organisation object structure. Without the indexing queue the switchboard queue becomes also superfluous. It has been removed as well.
- Removing the switchboard queue requires that all servlets are called without a opaque generic ('<?>'). That caused that all serlets had to be modified.
- Many servlets displayed the indexing queue or the size of that queue. In the past months the indexer was so fast that mostly the indexing queue appeared empty, so there was no use of it any more. Because the queue has been removed, the display in the servlets had also to be removed.
- The surrogate work task had been a part of the indexing queue control structure. Without the indexing queue the surrogates needed its own task management. That has been integrated here.
- Because the indexing queue had a special queue entry object and properties attached to this object, the propterties had to be moved to the queue entry object which is part of the new indexing queue withing the blocking queue, the Response Object. That object has now also the new properties of the removed indexing queue entry object.

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6225 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-07-17 13:59:21 +00:00
orbiter
597393db3b changed default visibility of classes/objects in upnp lib 
(eclipse tells me that this would improve performance,
 however, this removes compiler warnings)

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6224 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-07-16 12:19:40 +00:00
orbiter
eea4c17ef2 removed rpm parser 
- no-one used that thing
- loading huge rpm files bay be causes for crashes


git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6223 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-07-16 11:06:49 +00:00
orbiter
b332dfad67 - inserted request object into response object which carries this now instead generating new objects 
- fixed a problem with the crawler introduced in SVN 6216

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6222 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-07-15 23:08:35 +00:00
orbiter
ca72ed7526 -removed superfluous crawl cache 
-refactoring of crawler classes

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6221 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-07-15 21:07:46 +00:00
orbiter
8103ccec4c removed compiler warnings in imported classes 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6220 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-07-15 20:44:23 +00:00
lotus
52e371b8f7 suppress warnings for upnplib code 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6219 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-07-15 16:22:56 +00:00
lotus
477807e0e6 * updated jxpath to latest v1.3 
* added upnplib as source
	without packages:
	jmx
	remote
	samples

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6218 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-07-15 16:13:24 +00:00
orbiter
13c63f4082 a set of small fixes to crawling behaviour 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6216 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-07-15 14:15:51 +00:00
orbiter
a564df3984 update to mime types in parsers and httpd.mime 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6215 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-07-15 14:10:29 +00:00
orbiter
43c8defd79 enhanced parser with more extension + mime attributes 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6214 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-07-14 13:32:53 +00:00
orbiter
aee35bff6f replaced StringBuffer with StringBuilder in tar lib 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6213 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-07-14 13:31:57 +00:00
orbiter
49bbb9bd45 replaced tar library with integrated apache ant tar lib 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6212 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-07-14 11:31:40 +00:00
orbiter
f987fc6b4a added tar classes from apache ant tools 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6211 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-07-14 11:25:40 +00:00
orbiter
b2263bc720 enhanced document type recognition 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6209 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-07-14 11:01:05 +00:00
lotus
aa38eb5a20 * maxfilesize -1 for infinite filesize 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6208 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-07-14 08:39:39 +00:00
lotus
9cfe89c8fc * process content-length as soon as it is received 
* corrected indentation

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6206 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-07-13 19:55:13 +00:00
orbiter
50cf80056f removed jmimemagic library 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6203 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-07-13 10:58:37 +00:00
orbiter
3f113f38a8 removed unused imports 
removed unused libs from eclipse class path

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6201 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-07-13 10:19:10 +00:00
lotus
9f083bb6b2 check filetype before loading (no more mp4 loading) 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6200 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-07-12 16:50:11 +00:00
f1ori
076ae02c44 * added pl and py to extensions excepted by htmlParser 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6198 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-07-12 16:35:35 +00:00
f1ori
d5e51cfd09 * workaround for non-working build property replacements 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6197 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-07-12 09:38:03 +00:00
f1ori
f814e0fa81 enable warnings and fix most of it 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6196 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-07-11 21:01:27 +00:00
f1ori
8931c8d6b4 improvments to debianpackage: 
* autoupdate completely disabled, display hint
* restart-button in interface works!

* moved all build-Variables to yacyBuildProperties
* fixed some warnings


git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6195 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-07-11 17:03:22 +00:00
low012
fc1dc38b55 *) added spaces to make sure that no words are concatinated by accident 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6194 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-07-10 22:51:53 +00:00
low012
f242e7d7bc *) using Apache POI library to parse Word documents now 
*) removed tm-extractors library (can be found at http://www.textmining.org/ if necessary again)

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6193 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-07-10 22:49:16 +00:00
orbiter
caedd72400 - enhanced logging and exception details for parsers 
- removed inconsistencies in mime type declaration (one mime type should only appear once in all parsers)

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6192 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-07-10 16:15:33 +00:00
orbiter
4b74ad0a46 fixed setting of parser configuration servlets 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6191 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-07-10 15:02:34 +00:00
orbiter
57a88d435b redesign of parser mime type detection and parser steering 
There is now a mime-blacklist instead of a mime-whitelist

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6190 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-07-10 14:22:17 +00:00
lotus
e15d27bc63 avoiding double/wrong parser errors 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6189 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-07-10 09:31:10 +00:00
orbiter
21b8704fb4 refactoring of the ParserDispatcher and ParserConfig: resulted into Idiom, Parser and Classification classes 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6188 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-07-09 22:25:31 +00:00
orbiter
8ca1f5d400 - some work to integrate the html parser the same way as the other parsers are integrated (not finished) 
- added migration of code of settings pages (hmm.. does not work correctly yet, sorry)
- more refactoring
- removed more unused code

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6187 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-07-09 20:56:30 +00:00
low012
1ee109761f *) added changes which were lost 
*) additional annotations
*) additional svn properties
*) _no_ functional changes

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6186 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-07-09 20:13:11 +00:00
orbiter
499723891d removed all non-http daemons; they had not been used and may be a potential security risk. 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6185 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-07-08 22:24:34 +00:00
orbiter
0e8647d62f refactoring of search classes 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6184 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-07-08 22:14:57 +00:00
orbiter
dafffd0153 refactoring of parsers and document processing 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6182 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-07-08 21:48:08 +00:00
low012
8041e91f56 *) Ooops! 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6181 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-07-08 20:58:47 +00:00
low012
69551ff3d9 *) added several MIME types (derived from http://filext.com/), some of them might be rather uncommon 
*) added an annotation forgotten in last commit

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6180 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-07-08 20:53:08 +00:00
low012
11dfb2d54f minor changes: 
*) added annotations
*) set svn properties and added keywords to comments of parser classes
*) made a variable final to prevent (theoretical case of) change of object instance in synchronized block

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6179 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-07-08 19:20:03 +00:00
orbiter
77d2a3782c removed strange debugging strings 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6177 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-07-06 15:21:43 +00:00
lotus
4320f69574 universal handling for crashed parsers 
reverting r6090/1

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6176 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-07-05 08:12:14 +00:00
orbiter
024744245c small refactoring to prepare for new queues 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6173 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-07-04 12:17:10 +00:00
orbiter
16efcd0366 fix for http://forum.yacy-websuche.de/viewtopic.php?f=6&t=2252&hilit=&p=16389#p16389 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6172 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-07-04 06:24:58 +00:00
f1ori
0f3246e90a * fix debian package 
* add Class containing buildvariables


git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6171 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-07-03 21:57:05 +00:00
f1ori
8544cfd5a6 * remove seperate build-files for parsers 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6170 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-07-03 20:06:06 +00:00
orbiter
24cb6d68bc - renamed Stack to RecordStack to avoid name confusion with new classes 
- added new Stack class that implements a stack on BLOB files
- added new Stacks class that can be used for a set of Stacks (a 'Stack Database')
- added methods to other classes to support the new stacks

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6169 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-07-03 16:35:34 +00:00
orbiter
995da28c73 all stack/heap files that had been stored in DATA/PLASMA are now stored in the network-specific QUEUES path 
There is no migration. All crawls must be restarted.

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6167 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-07-02 17:01:23 +00:00
orbiter
aac89bf8ca trying to avoid "exceeding limit" message of server 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6166 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-07-02 15:47:42 +00:00
f1ori
48d78166ed * fix double copy of libraries 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6164 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-07-02 12:33:47 +00:00
lotus
7f868ca3c2 resource observer: support for yacyroot\DATA on an NTFS hardlink (Windows) 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6162 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-07-02 10:02:20 +00:00
orbiter
409538e17a code cleanup and code simplifcation 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6161 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-06-30 22:20:55 +00:00
orbiter
160031758d fix for problem with initializer 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6160 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-06-30 21:57:32 +00:00
orbiter
302a02cec8 moved all libraries from libx to lib 
removed libx directory
all libraries are now in lib, instead the test libraries in libt which are not part of releases

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6157 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-06-30 14:55:23 +00:00
orbiter
1f1399e5c5 extending visibility of objects and methods to avoid synthetic accessor methods and increase performance 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6156 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-06-30 13:25:46 +00:00
orbiter
154bbc3364 code cleanup: call of static methods directly to the class 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6155 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-06-30 13:01:35 +00:00
orbiter
222850414e simplification of the code: removed unused classes, methods and variables 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6154 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-06-30 09:27:46 +00:00
orbiter
93dfb51fd4 problems with code style 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6153 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-06-29 22:22:35 +00:00
orbiter
adf01c676e reduce lookup time when merging a large number of BLOBs 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6152 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-06-29 16:07:10 +00:00
orbiter
9a674d8047 - After the removal of the Tree class some code simplifications are possible. This affects mostly the Records class, which can be refactored and the result of the refactoring results in a reduced number of classes. 
- The EcoTable was renamed to Table.


git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6151 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-06-28 21:37:37 +00:00
orbiter
c5122d6836 completed migration of BLOBTree to BLOBHeaps: 
- removed migration code
- removed BLOBTree
after the removal of the BLOBTree, a lot of dead code appeared:
- removed dead code that was needed for BLOBTree
Some more classes may have not much use any more after the removal of BLOBTree, but still have some component that are needed elsewhere. Additional Refactoring steps are needed to clean up dependencies and then more code may appear that is unused and can be removed as well.

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6150 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-06-28 21:02:56 +00:00
orbiter
d1083a6913 maybe we have less problems with open connections to the server if we don't do BF forced sleeps (just a test) 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6149 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-06-28 10:35:12 +00:00
low012
ebe6c823ac *) changed svn properties agains (hopefully doing it right this time) 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6147 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-06-27 09:56:01 +00:00
low012
a80ac3a415 *) fixed wrong parser descriptions 
*) changed svn properties

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6146 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-06-27 09:42:33 +00:00
low012
457b6c0d6d *) updated Apache POI library to be able to parse Visio files 
*) updated PPT and XLS parsers to use new Apache POI library
*) added new Visio (VSD) parser

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6145 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-06-27 09:33:09 +00:00