Marown/yacy_search_server
1
0
Fork 0
mirror of https://github.com/yacy/yacy_search_server.git synced 2024-09-22 00:00:59 +02:00
Code Issues Actions Packages Projects Releases Wiki Activity
6,244 Commits 1 Branch 20 Tags 836 MiB
eb79ceb3ff
Commit Graph

3986 Commits

Author SHA1 Message Date
orbiter
cf739edc2e fix for possible deadlock, see 
http://forum.yacy-websuche.de/viewtopic.php?p=17017#p17017

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6252 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-08-07 12:11:22 +00:00
orbiter
6354b5e447 removed possible deadlock, see 
http://forum.yacy-websuche.de/viewtopic.php?p=17017#p17017

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6251 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-08-07 12:04:14 +00:00
orbiter
5cc17ccf8a a better caching with less overhead and more appropriate 
synchronisation use in more than 10 different data objects

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6250 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-08-07 11:55:32 +00:00
orbiter
92edd24e70 fixed problem with switching of networks 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6247 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-07-30 15:49:23 +00:00
orbiter
0575f12838 fix for deadlock 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6246 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-07-30 09:08:44 +00:00
orbiter
fbfdaf063d - patch to omit IndexOutOfBoundsException when a b64-encoded key appears not to be well-formed. In that case the key is still accepted but rated higher than other regular keys to create a virtual ordering between well-formed and ill-formed keys 
- check routine at the beginning of the import of table keys that check that all imported keys are well-formed. All records that have a ill-formed key are deleted. This is a hack and is not tested since I don't have bad data here to test with. If the effect is seen in the wild, please report in the forum.


git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6245 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-07-29 19:43:11 +00:00
orbiter
c0e17de2fb - fixes for some problems with the new crawling/caching strategies 
- speed enhancements for the cache-only cache policy by using special no-delay rules in the balancer
- fixed some deadlock- and 100% CPU problems in the balancer

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6243 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-07-25 21:38:57 +00:00
orbiter
634a01a9a4 replaced wget-requests with caching requests 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6242 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-07-24 14:52:27 +00:00
orbiter
c6c97f23ad - added cache usage properties to crawl start 
- added special rule to balancer to omit forced delays if cache is used exclusively
- extended the htCache size by default to 32GB

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6241 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-07-24 11:54:04 +00:00
orbiter
c4ae2cd03f fixed bug that caused deletion of crawl profiles at every application startup 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6240 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-07-23 22:09:02 +00:00
orbiter
161d2fd2ef redesign of access to the HTCache (now http.client.Cache): 
- better control to the cache by using combined request-header and content access methods
- refactoring of many classes to comply to this new access method
- make shure that the cache is always written if something was loaded
- some redesign of the process how http response results are feeded into the new indexing queue
- introduction of a cache read policy:
 * never use the cache
 * use the cache if entry exist
 * use the cache if the proxy freshness rule confirmes
 * use only the cache and go never online
- added configuration options for the crawl profiles to use the new cache policies. There is not yet a input during crawl start to set the policy but this will be added in another step.
- set the default policies for the existing crawl profiles. If you want them to appear in your default profiles you must delete the crawl profiles database; othervise the policy is 'proxy freshness rule'
- enhanced some cache access methods in such a way that unnecessary retrievals are omitted (i.e. for size computation). That should reduce some IO but also a lot of CPU computation because sizes were computed after decompression of content after retrieval of the content from the disc.

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6239 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-07-23 21:31:51 +00:00
f1ori
ba2e6de538 fix empty version string again 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6236 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-07-21 19:56:40 +00:00
orbiter
51534df0cb fix for possible synchronization problem 
see also: http://forum.yacy-websuche.de/viewtopic.php?f=6&t=2292&hilit=&p=16787#p16787

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6234 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-07-20 08:21:17 +00:00
orbiter
4da9042e8a code simplification 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6233 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-07-19 21:59:29 +00:00
orbiter
1d8d51075c refactoring: 
- removed the plasma package. The name of that package came from a very early pre-version of YaCy, even before YaCy was named AnomicHTTPProxy. The Proxy project introduced search for cache contents using class files that had been developed during the plasma project. Information from 2002 about plasma can be found here:
http://web.archive.org/web/20020802110827/http://anomic.de/AnomicPlasma/index.html
We stil have one class that comes mostly unchanged from the plasma project, the Condenser class. But this is now part of the document package and all other classes in the plasma package can be assigned to other packages.
- cleaned up the http package: better structure of that class and clean isolation of server and client classes. The old HTCache becomes part of the client sub-package of http.
- because the plasmaSwitchboard is now part of the search package all servlets had to be touched to declare a different package source.

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6232 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-07-19 20:37:44 +00:00
f1ori
67da20647f * add new odf parser based on sax-xml-parser 
* remove odf_utils-jar
* test metadata in ParserTest


git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6231 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-07-18 15:04:34 +00:00
f1ori
6d0e6d591b * ops, fix compiler error :( 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6227 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-07-17 20:02:56 +00:00
f1ori
3e5beb1654 * fix for empty version in seedlist 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6226 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-07-17 19:16:26 +00:00
orbiter
5bb8074150 removed the indexing queue. This queue was superfluous since the introduction of the blocking queues last year, where documents are parsed, analysed and stored in the index with concurrency. 
- The indexing queue was a historic data structure that was introduced at the very beginning at the project as a part of the switchboard organisation object structure. Without the indexing queue the switchboard queue becomes also superfluous. It has been removed as well.
- Removing the switchboard queue requires that all servlets are called without a opaque generic ('<?>'). That caused that all serlets had to be modified.
- Many servlets displayed the indexing queue or the size of that queue. In the past months the indexer was so fast that mostly the indexing queue appeared empty, so there was no use of it any more. Because the queue has been removed, the display in the servlets had also to be removed.
- The surrogate work task had been a part of the indexing queue control structure. Without the indexing queue the surrogates needed its own task management. That has been integrated here.
- Because the indexing queue had a special queue entry object and properties attached to this object, the propterties had to be moved to the queue entry object which is part of the new indexing queue withing the blocking queue, the Response Object. That object has now also the new properties of the removed indexing queue entry object.

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6225 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-07-17 13:59:21 +00:00
orbiter
eea4c17ef2 removed rpm parser 
- no-one used that thing
- loading huge rpm files bay be causes for crashes


git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6223 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-07-16 11:06:49 +00:00
orbiter
b332dfad67 - inserted request object into response object which carries this now instead generating new objects 
- fixed a problem with the crawler introduced in SVN 6216

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6222 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-07-15 23:08:35 +00:00
orbiter
ca72ed7526 -removed superfluous crawl cache 
-refactoring of crawler classes

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6221 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-07-15 21:07:46 +00:00
orbiter
13c63f4082 a set of small fixes to crawling behaviour 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6216 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-07-15 14:15:51 +00:00
orbiter
a564df3984 update to mime types in parsers and httpd.mime 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6215 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-07-15 14:10:29 +00:00
orbiter
43c8defd79 enhanced parser with more extension + mime attributes 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6214 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-07-14 13:32:53 +00:00
orbiter
49bbb9bd45 replaced tar library with integrated apache ant tar lib 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6212 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-07-14 11:31:40 +00:00
orbiter
b2263bc720 enhanced document type recognition 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6209 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-07-14 11:01:05 +00:00
lotus
aa38eb5a20 * maxfilesize -1 for infinite filesize 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6208 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-07-14 08:39:39 +00:00
lotus
9cfe89c8fc * process content-length as soon as it is received 
* corrected indentation

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6206 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-07-13 19:55:13 +00:00
orbiter
50cf80056f removed jmimemagic library 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6203 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-07-13 10:58:37 +00:00
orbiter
3f113f38a8 removed unused imports 
removed unused libs from eclipse class path

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6201 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-07-13 10:19:10 +00:00
lotus
9f083bb6b2 check filetype before loading (no more mp4 loading) 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6200 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-07-12 16:50:11 +00:00
f1ori
076ae02c44 * added pl and py to extensions excepted by htmlParser 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6198 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-07-12 16:35:35 +00:00
f1ori
d5e51cfd09 * workaround for non-working build property replacements 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6197 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-07-12 09:38:03 +00:00
f1ori
f814e0fa81 enable warnings and fix most of it 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6196 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-07-11 21:01:27 +00:00
f1ori
8931c8d6b4 improvments to debianpackage: 
* autoupdate completely disabled, display hint
* restart-button in interface works!

* moved all build-Variables to yacyBuildProperties
* fixed some warnings


git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6195 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-07-11 17:03:22 +00:00
low012
fc1dc38b55 *) added spaces to make sure that no words are concatinated by accident 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6194 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-07-10 22:51:53 +00:00
low012
f242e7d7bc *) using Apache POI library to parse Word documents now 
*) removed tm-extractors library (can be found at http://www.textmining.org/ if necessary again)

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6193 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-07-10 22:49:16 +00:00
orbiter
caedd72400 - enhanced logging and exception details for parsers 
- removed inconsistencies in mime type declaration (one mime type should only appear once in all parsers)

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6192 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-07-10 16:15:33 +00:00
orbiter
4b74ad0a46 fixed setting of parser configuration servlets 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6191 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-07-10 15:02:34 +00:00
orbiter
57a88d435b redesign of parser mime type detection and parser steering 
There is now a mime-blacklist instead of a mime-whitelist

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6190 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-07-10 14:22:17 +00:00
lotus
e15d27bc63 avoiding double/wrong parser errors 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6189 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-07-10 09:31:10 +00:00
orbiter
21b8704fb4 refactoring of the ParserDispatcher and ParserConfig: resulted into Idiom, Parser and Classification classes 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6188 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-07-09 22:25:31 +00:00
orbiter
8ca1f5d400 - some work to integrate the html parser the same way as the other parsers are integrated (not finished) 
- added migration of code of settings pages (hmm.. does not work correctly yet, sorry)
- more refactoring
- removed more unused code

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6187 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-07-09 20:56:30 +00:00
low012
1ee109761f *) added changes which were lost 
*) additional annotations
*) additional svn properties
*) _no_ functional changes

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6186 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-07-09 20:13:11 +00:00
orbiter
499723891d removed all non-http daemons; they had not been used and may be a potential security risk. 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6185 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-07-08 22:24:34 +00:00
orbiter
0e8647d62f refactoring of search classes 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6184 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-07-08 22:14:57 +00:00
orbiter
dafffd0153 refactoring of parsers and document processing 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6182 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-07-08 21:48:08 +00:00
low012
8041e91f56 *) Ooops! 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6181 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-07-08 20:58:47 +00:00
low012
69551ff3d9 *) added several MIME types (derived from http://filext.com/), some of them might be rather uncommon 
*) added an annotation forgotten in last commit

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6180 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-07-08 20:53:08 +00:00
low012
11dfb2d54f minor changes: 
*) added annotations
*) set svn properties and added keywords to comments of parser classes
*) made a variable final to prevent (theoretical case of) change of object instance in synchronized block

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6179 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-07-08 19:20:03 +00:00
orbiter
77d2a3782c removed strange debugging strings 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6177 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-07-06 15:21:43 +00:00
lotus
4320f69574 universal handling for crashed parsers 
reverting r6090/1

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6176 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-07-05 08:12:14 +00:00
orbiter
024744245c small refactoring to prepare for new queues 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6173 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-07-04 12:17:10 +00:00
orbiter
16efcd0366 fix for http://forum.yacy-websuche.de/viewtopic.php?f=6&t=2252&hilit=&p=16389#p16389 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6172 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-07-04 06:24:58 +00:00
f1ori
0f3246e90a * fix debian package 
* add Class containing buildvariables


git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6171 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-07-03 21:57:05 +00:00
f1ori
8544cfd5a6 * remove seperate build-files for parsers 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6170 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-07-03 20:06:06 +00:00
orbiter
24cb6d68bc - renamed Stack to RecordStack to avoid name confusion with new classes 
- added new Stack class that implements a stack on BLOB files
- added new Stacks class that can be used for a set of Stacks (a 'Stack Database')
- added methods to other classes to support the new stacks

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6169 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-07-03 16:35:34 +00:00
orbiter
995da28c73 all stack/heap files that had been stored in DATA/PLASMA are now stored in the network-specific QUEUES path 
There is no migration. All crawls must be restarted.

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6167 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-07-02 17:01:23 +00:00
orbiter
aac89bf8ca trying to avoid "exceeding limit" message of server 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6166 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-07-02 15:47:42 +00:00
f1ori
48d78166ed * fix double copy of libraries 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6164 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-07-02 12:33:47 +00:00
lotus
7f868ca3c2 resource observer: support for yacyroot\DATA on an NTFS hardlink (Windows) 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6162 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-07-02 10:02:20 +00:00
orbiter
409538e17a code cleanup and code simplifcation 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6161 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-06-30 22:20:55 +00:00
orbiter
160031758d fix for problem with initializer 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6160 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-06-30 21:57:32 +00:00
orbiter
302a02cec8 moved all libraries from libx to lib 
removed libx directory
all libraries are now in lib, instead the test libraries in libt which are not part of releases

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6157 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-06-30 14:55:23 +00:00
orbiter
1f1399e5c5 extending visibility of objects and methods to avoid synthetic accessor methods and increase performance 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6156 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-06-30 13:25:46 +00:00
orbiter
154bbc3364 code cleanup: call of static methods directly to the class 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6155 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-06-30 13:01:35 +00:00
orbiter
222850414e simplification of the code: removed unused classes, methods and variables 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6154 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-06-30 09:27:46 +00:00
orbiter
93dfb51fd4 problems with code style 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6153 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-06-29 22:22:35 +00:00
orbiter
adf01c676e reduce lookup time when merging a large number of BLOBs 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6152 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-06-29 16:07:10 +00:00
orbiter
9a674d8047 - After the removal of the Tree class some code simplifications are possible. This affects mostly the Records class, which can be refactored and the result of the refactoring results in a reduced number of classes. 
- The EcoTable was renamed to Table.


git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6151 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-06-28 21:37:37 +00:00
orbiter
c5122d6836 completed migration of BLOBTree to BLOBHeaps: 
- removed migration code
- removed BLOBTree
after the removal of the BLOBTree, a lot of dead code appeared:
- removed dead code that was needed for BLOBTree
Some more classes may have not much use any more after the removal of BLOBTree, but still have some component that are needed elsewhere. Additional Refactoring steps are needed to clean up dependencies and then more code may appear that is unused and can be removed as well.

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6150 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-06-28 21:02:56 +00:00
orbiter
d1083a6913 maybe we have less problems with open connections to the server if we don't do BF forced sleeps (just a test) 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6149 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-06-28 10:35:12 +00:00
low012
ebe6c823ac *) changed svn properties agains (hopefully doing it right this time) 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6147 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-06-27 09:56:01 +00:00
low012
a80ac3a415 *) fixed wrong parser descriptions 
*) changed svn properties

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6146 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-06-27 09:42:33 +00:00
low012
457b6c0d6d *) updated Apache POI library to be able to parse Visio files 
*) updated PPT and XLS parsers to use new Apache POI library
*) added new Visio (VSD) parser

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6145 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-06-27 09:33:09 +00:00
apfelmaennchen
a10c8022d1 DidYouMean: 
- limit the number of consumer threads to available CPUs
- added some javadoc

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6144 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-06-27 07:23:34 +00:00
f1ori
7eb3bff5b3 * workaround for http://forum.yacy-websuche.de/viewtopic.php?f=6&t=2220&hilit=#p16128 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6143 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-06-26 14:05:39 +00:00
orbiter
99fa265e1d fix for search bug caused by tenant patch 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6125 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-06-22 22:31:29 +00:00
orbiter
79875782af be a bit more lazy when removing domain navigation entries 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6120 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-06-22 12:30:22 +00:00
orbiter
57af311627 fix for wrong urls in navigator when a tenant is used 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6119 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-06-22 12:25:18 +00:00
lotus
76b96337e2 just some chatty code 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6118 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-06-22 11:45:21 +00:00
low012
91785d895c *) minor changes in comments 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6109 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-06-21 20:08:52 +00:00
orbiter
bdda140c02 fix for json output (no doubleqotes any more, doublequote quoting did not work) 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6105 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-06-20 23:33:29 +00:00
orbiter
2f84736120 ignore signature files that cannot be downloaded because of failed encoding 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6103 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-06-20 20:46:58 +00:00
orbiter
041d9c253e some refactoring and more error-awareness in LogalizeHandler 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6102 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-06-20 20:29:57 +00:00
orbiter
6b307d6d59 more tolerance for corrupted index entries in exported row sets 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6099 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-06-19 21:35:44 +00:00
orbiter
33aafa9b4b better logging when writing merged dumps 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6098 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-06-19 17:02:50 +00:00
lotus
db70badcf0 possibility to set remote host on upnp device 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6097 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-06-19 15:32:59 +00:00
orbiter
4d29e90708 uaeh 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6096 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-06-19 14:54:39 +00:00
orbiter
3c3e6499ae added more logging for merge operation 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6095 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-06-19 14:51:35 +00:00
orbiter
15180fc95e - patch for future computation in SplitTable 
- added same concurrent process for has() from SPlitTable in ArrayStack

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6093 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-06-18 23:24:23 +00:00
orbiter
9a5ec20b3c avoid merge during startup 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6092 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-06-18 21:18:56 +00:00
lotus
bf6b92343c try to avoid stuck pdf parser 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6091 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-06-18 06:31:55 +00:00
lotus
c695c7f512 try to remove hung swf parser from queue 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6090 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-06-17 17:48:02 +00:00
orbiter
fc69a76197 update to web structure picture: 
- allow bigger size
- better instructions for api usage

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6089 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-06-17 12:37:31 +00:00
orbiter
ae015e8e98 refactoring of blob package classes 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6088 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-06-17 09:58:15 +00:00
orbiter
8b8877c233 moved image collector 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6087 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-06-16 21:48:09 +00:00
orbiter
be1c7ddc64 refactoring of search classes -- moved Ranking Profile to search package 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6086 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-06-16 21:45:40 +00:00
orbiter
fd31a3616a - more logging in server process 
- fix for bas ascii in comment

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6084 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-06-16 15:10:59 +00:00
orbiter
5a7fd6b4c8 just some comment lines 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6081 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-06-16 06:38:26 +00:00
orbiter
31f60a3b3e when doing searches, also apply a online caution to DHT transmission and stop transmissions while heavy load caused by searching. This omits the many requests to the URL database that are needed for DHT transfer and it avoids collisions with URL retrieval needed for search results. 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6080 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-06-15 22:03:27 +00:00
orbiter
17dc6d4be5 small fix for new Logger 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6079 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-06-15 21:39:24 +00:00
orbiter
ce1adf9955 serialized all logging using concurrency: 
high-performance search query situations as seen in yacy-metager integration showed deadlock situation caused by synchronization effects inside of sun.java code. It appears that the logger is not completely safe against deadlock situations in concurrent calls of the logger. One possible solution would be a outside-synchronization with 'synchronized' statements, but that would further apply blocking on all high-efficient methods that call the logger. It is much better to do a non-blocking hand-over of logging lines and work off log entries with a concurrent log writer. This also disconnects IO operations from logging, which can also cause IO operation when a log is written to a file. This commit not only moves the logger from kelondro to yacy.logging, it also inserts the concurrency methods to realize non-blocking logging.

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6078 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-06-15 21:19:54 +00:00
lotus
aec3e7995a autoconfig.pac can be used to browse .yacy-domains only 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6077 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-06-15 19:48:11 +00:00
orbiter
bc6dd8194b refactoring: moved search query class to new search package 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6075 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-06-15 11:49:00 +00:00
orbiter
a4805defdd added stub for new search process 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6074 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-06-15 11:46:23 +00:00
orbiter
b8e738a7be a collection of 
- small bug fixes
- better/more comments
- more asserts
- fixed synchronization
- test case enhancements
- code cleanup
- performance hacks

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6073 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-06-14 22:09:08 +00:00
apfelmaennchen
39779e4796 DidYouMean: as I moved to only 8 consumer and 4 producer threads, I removed poison pills as it does not make sense anymore - threads are interrupted directly. Having a consumer thread per test case just didn't make sense either (see svn 6070) due to the massive overhead. 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6072 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-06-14 16:31:31 +00:00
apfelmaennchen
c3c4dd0933 DidYouMean - changed to much simpler LinkedBlockingQueue 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6071 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-06-14 15:25:57 +00:00
apfelmaennchen
01ac1b5d7e - blocking queue implementation of DidYouMean 
- timeout ist set to 500ms

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6070 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-06-14 11:53:09 +00:00
orbiter
b8bb1bb364 join with a timeout does not cause that the corresponding thread is stopped after the time-out. It does only cause that the waiting is stopped. Here we need additionally a signal to the thread to stop after we finished waiting. 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6069 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-06-13 23:54:52 +00:00
orbiter
b69f22e9ca mistake in last commit: computation of loops in ReversingTwoConsecutiveLetters 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6068 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-06-13 23:37:51 +00:00
orbiter
3130334932 - start first with threads that run more loops 
- join first with threads that run less loops

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6067 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-06-13 23:34:16 +00:00
apfelmaennchen
6cde7ebf16 DidYouMean 
- without I/O intensive sorting by count
- but with multiple threads

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6066 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-06-13 23:16:14 +00:00
orbiter
f348190566 tried to insert a database dump import method to the phpBB3 import function. Reason: imports or large database dumps are cannot be handled with phpMyAdmin and this should be an easy way to the database dumps into a mySQL database where it can be exported again with the phpBB3 content integration adapter. Completion or removal of this function stub will follow before next main release. 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6065 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-06-13 23:03:40 +00:00
orbiter
945777aa80 replaced rwi term counting method by one that computes the maximum of the blobs that contibute to the RWI. An addition of the blob sizes is wrong/incorrect and does not reflect the real size. Truncation the size operation to the maximum of all blobs is also incorrect, but not as wrong as the sum of all blob sizes wich double-counts many rwi entries. 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6064 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-06-13 22:59:54 +00:00
orbiter
7c4d1d471c hand-over of more specific object 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6062 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-06-13 10:22:25 +00:00
apfelmaennchen
09acfa66d1 - improved "did you mean" 
- added &meanCount= to query string
- &meanCount=0 ==> no suggestion, no performance loss
- sorting suggestions by sb.indexSegment.termIndex().count()

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6059 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-06-13 06:20:05 +00:00
apfelmaennchen
da6ce37f7b - fixed encoding problem 
- added limit to 10 suggestions

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6058 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-06-12 21:36:26 +00:00
apfelmaennchen
54a48b4184 - added "did you mean" to search page 
- currently works for single word queries only!

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6057 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-06-12 20:36:03 +00:00
orbiter
0fc1168554 - reduced time-out for socket-connection communication from 20 seconds to 5 seconds. This is a test to find out if the time-out was a cause for problems in metager environments 
- turned a fine log entry in case of rejected connections on the server socket into a warning. (look for 'exceeding limit')


git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6051 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-06-11 10:20:31 +00:00
orbiter
28b86385cd patch for bad behaving swf parser 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6050 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-06-11 09:54:48 +00:00
orbiter
d58b395993 fix for http://forum.yacy-websuche.de/viewtopic.php?p=15693#p15693 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6049 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-06-11 09:38:25 +00:00
orbiter
733385cdd7 enahnced database access times by removal of unnecessary synchronization. 
added also more hacks that resulted from high-volum query testing

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6047 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-06-10 23:02:42 +00:00
orbiter
398e210fef removed synchronization in logging that causes deadlocks in high-performance environments 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6044 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-06-10 19:17:30 +00:00
orbiter
db3a06dd81 removed cookie handling in httpc: 
- no need to do cookie handling in proxy, this was switched off so far
- no need for cookies in crawler, this was switched on (by mistake)
This fix was needed for a case where a web server flooded the crawler with cookies and caused a complete blocking of the httpc.

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6043 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-06-10 16:11:09 +00:00
orbiter
1c54ae4a63 some small changes in HandleMap Testing 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6042 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-06-10 15:02:52 +00:00
orbiter
2c5554c912 small enhancements in search result computation speed 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6039 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-06-09 15:22:23 +00:00
orbiter
e0b3984805 added navigation keys for site and author facets to remote search interface 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6038 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-06-09 09:07:52 +00:00
orbiter
27fa6a66ad - completed the author navigation 
- removed some unused variables

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6037 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-06-08 23:30:12 +00:00
orbiter
a9a8b8d161 - added display of author navigation (usage of that navigator not yet implemented 
- added a synchronization in pdf parser which should help to avoid deadlocks that occur when displaying several search results pointing to pdf sources
- fixed smaller bugs in navigation

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6036 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-06-08 22:01:26 +00:00
orbiter
c879783008 added steering of navigator computation: 
- by default the navigator computation if off for servlet yacysearch.html, but:
- the servlet is called by default with a option to switch navigator results on
this will prevent that metasearch users will get slow results that are caused by unnecessary computations

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6035 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-06-07 22:51:15 +00:00
orbiter
c079b18ee7 - refactoring of IntegerHandleIndex and LongHandleIndex: both classes had been merged into the new HandleMap class, which handles (key<byte[]>,n-byte-long) pairs with arbitraty key and value length. This will be useful to get a memory-enhanced/minimized database table indexing. 
- added a analysis method that counts bytes that could be saved in case the new HandleMap can be applied in the most efficient way. Look for the log messages beginning with "HeapReader saturation": in most cases we could save about 30% RAM!
- removed the old FlexTable database structure. It was not used any more.
- removed memory statistics in PerformanceMemory about flex tables and node caches (node caches were used by Tree Tables, which are also not used any more)
- add a stub for a steering of navigation functions. That should help to switch off naviagtion computation in cases where it is not demanded by a client

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6034 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-06-07 21:48:01 +00:00
orbiter
bead0006da replaced tmp file extensions by prt 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6033 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-06-06 18:09:58 +00:00
orbiter
3189f9cd39 fixed problem with DCEntry initialization 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6032 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-06-06 18:00:50 +00:00
orbiter
a704d82280 patch for problem with digest 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6031 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-06-06 16:53:16 +00:00
orbiter
3029ef6eb3 fixed a bug that was recently inserted which caused that no idx and gap files were written. 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6030 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-06-06 16:43:58 +00:00
orbiter
b6e274f211 omit most of forced crawl delays by using a separat delay table which flushes delayed URLs at the correct time 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6029 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-06-06 16:20:27 +00:00
orbiter
d50be59088 - added a automatic re-construction of the domain stack after 10 minutes. this includes then urls to the domain stack that were left over in case of stack size limitations when the domain stack was created the last time 
- changed the busy sleep time for the crawl thread to 30 millisecons. This is sufficient to crawl with 2000 PPM.

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6028 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-06-06 09:34:44 +00:00
orbiter
5fdba0fa51 - fixed a not working selection rule in balancer 
- more security about crawl-delay, be more fail-save
- better logging in case of long forced crawl-delays

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6027 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-06-06 08:46:59 +00:00
orbiter
f5602404d5 another speed boost for the balancer 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6026 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-06-06 02:37:04 +00:00
orbiter
95e8cbd1c3 new fully redesigned balancer and bugfixes regarding lost profile handles and killed crawls 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6025 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-06-06 01:56:31 +00:00
orbiter
c062385552 fix for http://forum.yacy-websuche.de/viewtopic.php?p=15555#p15555 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6024 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-06-05 18:18:16 +00:00
orbiter
42ae40b9f6 some bugfixes to database close() methods 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6023 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-06-04 22:43:46 +00:00
orbiter
a0c53abbe1 - wait until local results are computed during search, see http://forum.yacy-websuche.de/viewtopic.php?f=5&t=2167&hilit=&p=15521#p15521 
- show only x+1 pages in page navigator

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6022 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-06-04 20:58:47 +00:00
orbiter
9bfd22f65d fix for http://forum.yacy-websuche.de/viewtopic.php?p=15523#p15523 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6020 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-06-04 19:57:25 +00:00
orbiter
1c77db670f re-designed response format for navigation: 
- changed json and rss response templates


git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6019 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-06-04 10:54:49 +00:00
orbiter
15fad767c0 some refactoring of topic generation 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6018 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-06-03 23:49:06 +00:00
orbiter
cc49aedf12 - fixed problem with remote search NPE 
- more abstraction for search requests

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6015 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-06-03 08:49:54 +00:00
f1ori
9e18abc2ac * fix charset detection, see http://forum.yacy-websuche.de/viewtopic.php?f=5&t=2137 
* why has this been uncommented???


git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6014 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-06-02 20:54:13 +00:00
orbiter
c38c852090 modified access method to get index entries out of a array of BLOBs: 
iterate them, then merge; not collect them and merge then.
This should use less memory and may behave better in an environment with many queries.
To ensure that too many queries will not cause total blocking,
a time-out of one second was also added. After the time-out
the index data that was collected so far is returned.

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6013 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-06-02 16:53:45 +00:00
orbiter
ab06a6edd2 renamed topwords to topics and enhanced computation methods of topics 
topics will now only be computed using the document title, not the document url,
because the host navigator is now responsible for statistical effects of urls.

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6011 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-06-02 15:20:10 +00:00
orbiter
a5d481eab1 enhanced navigation 
- fixed too early computation of navigation
- moved navigation rendering to yacysearchtrailer
- added more asserts

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6006 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-06-01 22:45:28 +00:00
orbiter
7639ec2f38 - fixed letter case bug for dc record creation 
- dc parser is now lazy against letter cases

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5998 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-05-29 15:09:37 +00:00
orbiter
4522c13ee7 added option for a table prefix when importing phpbb3 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5996 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-05-29 14:29:02 +00:00
orbiter
1c69d9b8b6 more refactoring of the index classes 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5995 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-05-29 14:16:41 +00:00
orbiter
3d5f2ff544 - added new servlets to support search portal administrators for the integration of yacy search fields in their web pages 
- moved some servlets from here to there..
- changed menu structure
- removed yacyui-portaltest.html which contained an example for the live search which is now integrated on all pages in yacy. The code snippet example from that page is integrated into the ConfigLiveSearch.html servlet


git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5994 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-05-29 14:16:03 +00:00
orbiter
4d4315687f fix for problem with concurrency in host navigator, bug reported by wsb 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5993 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-05-29 10:52:50 +00:00
orbiter
88426912ad more refactoring to make the segment object easier to use and to be prepared to integrate author navigation 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5992 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-05-29 10:03:35 +00:00
lotus
d813fd26ed reset sent/received counters on index delete 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5991 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-05-28 15:49:42 +00:00
orbiter
99bf0b8e41 refactoring of plasmaWordIndex: 
divided that class into three parts:
- the peers object is now hosted by the plasmaSwitchboard
- the crawler elements are now in a new class, crawler.CrawlerSwitchboard
- the index elements are core of the new segment data structure, which is a bundle of different indexes for the full text and (in the future) navigation indexes and the metadata store. The new class is now in kelondro.text.Segment

The refactoring is inspired by the roadmap to create index segments, the option to host different indexes on one peer.

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5990 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-05-28 14:26:05 +00:00
orbiter
876746602d catch problems of file hash computation, see also: 
http://forum.yacy-websuche.de/viewtopic.php?p=15245#p15245

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5989 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-05-28 10:08:36 +00:00
orbiter
fec6f9054f some refactoring of search methods 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5988 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-05-27 23:51:34 +00:00
orbiter
3d4b826ca5 migration of all databases that use the deprecated BLOBTree format into the BLOBHeap format. Old databases are migrated automatically. 
This removes the last very IO-intensive data structures which were still used for Wiki, Blog and Bookmarks. Old database files will still remain in the DATA subdirectory but can be deleted manually if no major bugs appear during migration. There is no need for any user action, all migration is done automatically.

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5986 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-05-27 15:04:04 +00:00
orbiter
4b4bddca00 added new submenu to crawler menu: import of phpbb3 forum postings from mysql 
- yacy can import phpbb3 posts without crawling
- all data is written as surrogate
- indexed surrogate files can be re-used

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5985 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-05-27 14:53:23 +00:00
orbiter
d8284046b0 enhanced speed of site navigation computation 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5980 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-05-26 22:30:20 +00:00
orbiter
c72a5cf326 added stub for PHPBB3 extraction code using direct access to mySQL 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5979 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-05-26 15:58:49 +00:00
orbiter
e735d3a69f fix for http://forum.yacy-websuche.de/viewtopic.php?p=15175#p15175 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5978 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-05-26 15:03:50 +00:00
orbiter
63a0255166 - refactoring: added new content package, which will contain connector classes for different types of data sources to import texts into the YaCy index 
- refactoring: migrated data objects for the new connector classes
- added a DAO interface class to specify an abstract interface for database retrieval connector methods

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5977 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-05-26 07:44:22 +00:00
orbiter
f246928c20 first attempt to add 'real' Navigation to yacy search results: host navigation 
- after a search is started, it is analysed how many hits are in each site
- this can be done really efficient, because the navigation information is hidden in the url hash and can be computed very fast
- the search result shows a column on the right with the hosts and the hits per host
- after a click on a host the search is modified using the efficient site: - operator

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5976 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-05-25 22:27:34 +00:00
orbiter
54b9e99c01 - more information about peer tags 
- peer tag is by default '*'

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5975 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-05-25 21:43:33 +00:00
orbiter
26a46b5521 increased default maximum file size for database files to 2GB 
Other file sizes can now be configured with the attributes
filesize.max.win and filesize.max.other
the default maximum file size for non-windows OS is now 32GB

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5974 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-05-25 06:59:21 +00:00
orbiter
addecdb18c simplified code, removed one unused method in all implementing classes 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5972 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-05-21 23:53:01 +00:00
borg-0300
47fce9020c small change (Orbiter's wish) 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5971 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-05-21 17:51:52 +00:00
borg-0300
e07b14e5d7 finally a working fix for 5960 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5970 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-05-21 16:07:04 +00:00
borg-0300
3ebb904d2c fix for 5960, http://forum.yacy-websuche.de/viewtopic.php?f=6&t=2119 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5969 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-05-21 11:47:57 +00:00
lotus
734680dc70 initialize the ResourceObsever in own thread 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5968 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-05-21 08:30:34 +00:00
orbiter
e005cfea37 fix for bug in -incell option of URLAnalysis 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5967 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-05-21 06:57:03 +00:00
orbiter
a7e392f31b The collection index will not be supported any more. 
Existing indexes based on the old index collections must be migrated with YaCy 0.8
- removed index collection classes and all migration tools
- added a 'incell' reference collection feature in URL analysis


git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5966 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-05-20 14:51:26 +00:00
orbiter
a2f48863fc - added prototype for navigation index 
- refactoring of word index prototype
(no functional changes so far)

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5965 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-05-20 09:00:24 +00:00
lotus
47fd226bdb proper parsing of sentences 
does not affect tokens/words

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5964 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-05-19 16:41:27 +00:00
orbiter
27eb8d62cb - new development cycle 
- removed temporary configuration with safe setting for indexer threads (=1) and replaced it with best value computed during performance tests (1/2 of number of processors)

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5963 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-05-18 21:20:06 +00:00
orbiter
b7457d3807 patch for http://forum.yacy-websuche.de/viewtopic.php?p=14720#p14720 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5960 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-05-17 21:44:02 +00:00
orbiter
bffbe43e09 fix for http://forum.yacy-websuche.de/viewtopic.php?p=14522#p14522 
fix for http://forum.yacy-websuche.de/viewtopic.php?p=14955#p14955

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5959 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-05-17 21:15:06 +00:00
orbiter
f133d6065c fix for http://forum.yacy-websuche.de/viewtopic.php?p=14955#p14955 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5958 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-05-17 18:28:33 +00:00
lotus
82af994041 added missing loglevel 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5956 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-05-16 08:21:51 +00:00
orbiter
ad9762746d no exception in case of uniq() time-out, see also 
http://forum.yacy-websuche.de/viewtopic.php?p=13177#p13177

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5955 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-05-15 23:07:10 +00:00
orbiter
1efe686e3f fix for http://forum.yacy-websuche.de/viewtopic.php?p=13960#p13960 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5954 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-05-15 22:51:15 +00:00
lotus
13fb84ab81 you can define your default number of search results displayed by search.items 
this applies only to requests through the classic-style page

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5953 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-05-15 14:48:34 +00:00
orbiter
f2e4d156e8 removed debug messages 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5950 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-05-12 22:03:33 +00:00
orbiter
709bfc2cd4 added a memory check in http post protocol 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5949 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-05-12 20:23:55 +00:00
orbiter
c01d6f43e1 - fixed problem with thread dump if no arguments are given 
- rejecting peers that are older than 6 hours (not-seen during 6 hours)
- 0.78, targeting 0.8 at the end of the week

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5948 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-05-11 22:26:21 +00:00
orbiter
a49edd9415 fix for bug in search with site: constraint 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5947 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-05-11 21:20:23 +00:00
orbiter
c1e5fad9a7 fix for http://forum.yacy-websuche.de/viewtopic.php?p=14767#p14767 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5944 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-05-10 20:50:46 +00:00
orbiter
8ee3a94e82 fix for non-caching of sitehash, see http://forum.yacy-websuche.de/viewtopic.php?p=14440#p14440 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5942 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-05-10 11:44:17 +00:00
borg-0300
21930d05ed fix for [B@... 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5941 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-05-10 10:54:06 +00:00
orbiter
b6ba387e01 fix for http://forum.yacy-websuche.de/viewtopic.php?p=14751#p14751 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5940 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-05-10 10:46:26 +00:00
orbiter
4338dcf936 fix for http://forum.yacy-websuche.de/viewtopic.php?f=6&t=2093&hilit= 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5937 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-05-09 19:07:34 +00:00
lotus
bad7ce9286 experimental option trayIcon.force for unsupported platforms. java 1.6 needed 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5936 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-05-09 18:35:02 +00:00
low012
ea27853c59 *) some refactoring 
*) added one assertion
*) no functional changes

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5935 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-05-09 13:43:38 +00:00
low012
d164b42604 *) cosmetics 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5934 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-05-08 19:26:36 +00:00
orbiter
17150b2950 fixed bug in snippet computation 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5932 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-05-08 15:26:32 +00:00
orbiter
89aeb318d3 enhanced the wikimedia dump import process 
enhanced the wiki parser and condenser speed

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5931 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-05-08 10:36:13 +00:00
orbiter
5fb77116c6 added a submenu to index administration to import a wikimedia dump (i.e. a dump from wikipedia) into the YaCy index: see 
http://localhost:8080/IndexImportWikimedia_p.html

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5930 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-05-08 07:54:10 +00:00
hermens
df733af4fa Try not to loose content from ram during IndexCell.delete by moving ram.delete after the dangerous operations on the array (array.get and array.delete) 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5929 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-05-06 12:53:17 +00:00
hermens
ac72005f2f Let IndexCell.remove remove entries from the ram portion of the DB as well. 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5928 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-05-06 12:32:34 +00:00
orbiter
8ba7ff5353 a fix and another speed enhancement for the RWI cache 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5927 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-05-05 22:40:40 +00:00
orbiter
05f077e85f added stack trace output to solve problem in 
http://forum.yacy-websuche.de/viewtopic.php?f=6&t=2076&hilit=&p=14612#p14612

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5926 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-05-05 20:24:20 +00:00
orbiter
71a4cadf31 better and more performant synchronization in SimpleARC, the caching object for word hashes. Speeds up indexing. 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5925 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-05-05 20:19:51 +00:00
orbiter
e6773cbb33 better handling of RWI cache for concurrency and less overhead when writing new entries -> even more indexing speed 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5924 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-05-05 20:08:23 +00:00
orbiter
c097531e3d added a catch Exception to all thread to check if any of them silently dies without any other notification 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5922 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-05-05 06:31:35 +00:00
orbiter
083533e5ec fix for bugs in IODispatcher 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5921 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-05-04 21:37:59 +00:00
orbiter
21fbca0410 better scaling of HEAP dump writer for small memory configurations; 
should prevent OOMs during cache dumps

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5920 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-05-04 08:29:44 +00:00
orbiter
6e0b57284d better care for states of the IODispatcher 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5919 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-05-03 22:54:47 +00:00
orbiter
1db9cdd4e4 fixed bug in writing of robots.txt entries in case that host names exceeded 64 characters and some other problems 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5918 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-05-03 19:35:10 +00:00
f1ori
bde88b684a * splitt off yacyRelease from yacyVersion 
* added some gui infos about signatures


git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5916 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-05-02 12:12:22 +00:00
orbiter
057ce14c8e more fixes (character encoding, parser exceptions, http client failure, blob writing) 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5914 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-05-02 07:43:03 +00:00
orbiter
d2ac0aa682 - fixed possible bugs in Stack (may affect Crawler reset) and RandomAccess handling 
- increased default memory size to 180MB
- fixed possible bug in http client reset (there was a deadlock)
- bug in BOBHeap marked, but not solved, cause is still unknown.

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5912 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-05-02 01:40:03 +00:00
lotus
1351d903a1 don't follow links like mailto: 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5909 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-05-01 08:53:50 +00:00
orbiter
e88a66bcae temporary disabling computation of all sublinks (check needed) 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5908 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-05-01 07:30:53 +00:00
low012
ff5f82d780 *) removed description of removed commands from wikiHelp ([= =]) 
*) used format function of Netbeans for wikiCode to make it more readable, no functional changes

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5907 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-05-01 07:28:59 +00:00
orbiter
eacf95213a fix for crawling of mailto-links 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5906 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-05-01 07:25:55 +00:00
orbiter
9c6ac43f66 fixes for wiki parser 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5905 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-04-30 22:03:35 +00:00
orbiter
3a64c9d02f - fix for problem with concurrency when computing word hashes 
- fix for search in case that a urlfilter was used and zero results were returned

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5904 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-04-29 22:14:12 +00:00
orbiter
d3f8aa5a2a set of small fixes 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5903 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-04-29 21:36:20 +00:00
low012
78ffb61297 *) got rid of unnecessary variable which might also fix IndexOutOfBoundsException 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5902 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-04-29 21:08:44 +00:00
orbiter
d31e6f9c14 fix for http://forum.yacy-websuche.de/viewtopic.php?p=14457#p14457 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5899 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-04-29 11:18:17 +00:00
orbiter
8d6212233b fix for IODispatcher 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5896 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-04-28 07:24:28 +00:00
orbiter
f678472f46 fix for quote problem in json output 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5895 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-04-27 22:27:02 +00:00
orbiter
d079d6dfdb small changes in surrogate reader, wiki code and portal test 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5894 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-04-27 20:30:43 +00:00
orbiter
07f09742bb set of small fixes and comments 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5893 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-04-27 15:29:50 +00:00
borg-0300
06ed4ef7b3 * better picture handling 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5891 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-04-27 11:19:15 +00:00
orbiter
5a634cab23 removed generation of anchor link sets in document types that describe container formats. 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5890 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-04-27 08:46:11 +00:00
low012
f1244264b8 *) hopefully fixed bug reported in http://forum.yacy-websuche.de/viewtopic.php?t=2057 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5882 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-04-26 16:18:14 +00:00
orbiter
2e3186189b fix for mediawikiIndex surrogate producer + added concurrency 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5880 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-04-25 21:52:21 +00:00
apfelmaennchen
6f5ea7b1a8 small fix for previous post 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5879 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-04-25 21:28:08 +00:00
apfelmaennchen
138a0747e3 added serverObjects.putJSON as JSON has very particulare encoding requirements 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5877 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-04-25 20:56:29 +00:00
orbiter
d977dd9a96 fix for surrogate loader 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5870 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-04-24 22:54:40 +00:00
orbiter
9cb68353da fix for bug in ProfilingGraph for ppm >> 10000 ppm (!) 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5868 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-04-24 13:18:20 +00:00
orbiter
9e4db75aac reduced internal logging and reduced memory that internal logging can use 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5867 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-04-24 12:09:04 +00:00
orbiter
c10c257255 attempt to fix a deadlock situation where the IODispatcher did not work. 
I suspect the dispatcher thread has crashed and queues filled so no indexing process was able to write data.
This fix tries to heal the problem, but I am unsure if it helps. To get a better view of the problem, some more log outputs had been inserted.
Added also a new attribut indexer.threads to get a control over the number of default threads for the indexer (default is 1)


git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5866 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-04-24 11:55:39 +00:00
orbiter
09987e93fd fixed some more bad handling of byte[] 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5865 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-04-23 22:02:12 +00:00
orbiter
1bcc1450cb more explaining error message in case of IOExceptions during html parsing 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5864 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-04-23 21:18:01 +00:00
orbiter
fe51f4d668 less synchronization may help to prevent deadlocks 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5863 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-04-23 20:54:13 +00:00
orbiter
58802e4201 added missing success test in storeDocumentIndex, 
see http://forum.yacy-websuche.de/viewtopic.php?f=5&t=1922&hilit=

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5862 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-04-23 20:38:56 +00:00
orbiter
171e62bee5 addition to the fix from last commit (which did not work) 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5860 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-04-23 16:36:21 +00:00
orbiter
059949a0d1 tried to fix problem with snippet fetch for second search page when verify=false 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5859 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-04-23 15:29:30 +00:00
lotus
b08991e278 moved some constants, rename of Tray class 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5858 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-04-23 13:18:59 +00:00
orbiter
138422990a - removed useCell option: the indexCell data structure is now the default index structure; old collection data is still migrated 
- added some debugging output to balancer to find a bug
- removed unused classes for index collection handling
- changed some default values for the process handling: more memory needed to prevent OOM

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5856 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-04-22 22:39:12 +00:00