Marown/yacy_search_server
1
0
Fork 0
mirror of https://github.com/yacy/yacy_search_server.git synced 2024-09-21 00:00:13 +02:00
Code Issues Actions Packages Projects Releases Wiki Activity
4,065 Commits 1 Branch 20 Tags 836 MiB
64261754f4
Commit Graph

2718 Commits

Author SHA1 Message Date
orbiter
367fc28928 corrected Brausse->Brausze 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4020 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2007-08-01 22:15:51 +00:00
low012
ea05c7d463 *) characters are saved in ASCII-safe notation in language data files now ('\u0063' instead of 'a'), this hopefully avoids lots of problems 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4019 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2007-07-31 21:27:28 +00:00
orbiter
757703a938 synchronization of access tracker to avoid java-internal loop in TreeMap during shutdown 
see http://forum.yacy-websuche.de/viewtopic.php?p=1178#p1178

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4017 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2007-07-31 10:42:11 +00:00
orbiter
beb2e062a7 fix for http://forum.yacy-websuche.de/viewtopic.php?f=6&t=192&p=1129#p1129 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4016 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2007-07-31 10:17:39 +00:00
orbiter
e76fe1c078 - replaced unicode characters in copyright holder name ('Brausse') 
- more logging for bootstrap seedlist loading
- larger DHT chunks

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4015 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2007-07-31 10:00:17 +00:00
orbiter
31023dbc7a added package declaration for language package 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4014 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2007-07-30 22:53:51 +00:00
low012
255e754257 *) Removed unnecessary condition (always true). 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4013 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2007-07-30 21:03:07 +00:00
low012
a42d233944 *) Filename for output file is created from language code + .ldf now (e.g. en-GB.ldf) 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4012 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2007-07-30 21:00:42 +00:00
low012
ffc607d65d *) Fixed a small bug found by Lotus. 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4011 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2007-07-30 13:03:10 +00:00
low012
33e2423bde *) Tool to find out the percentage of letters in a text. First step to build a tool which detects the language a text is written in. 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4010 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2007-07-29 22:05:20 +00:00
low012
3ee9a26ef2 *) deleted unused program 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4009 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2007-07-29 15:30:29 +00:00
low012
d36786bf37 *) startYACY.sh is compatible with OpenSolaris now (see http://forum.yacy-websuche.de/viewtopic.php?t=203 for details) 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4007 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2007-07-27 21:56:49 +00:00
orbiter
8d6aa7a66d replaced detailed search page by ranking definition page (this is what it essentially is) 
the ranking definition there will influence the normal web search

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4006 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2007-07-25 14:04:36 +00:00
orbiter
7ff4357184 fix for http://forum.yacy-websuche.de/viewtopic.php?f=6&t=206&hilit=&p=1130#p1130 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4004 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2007-07-24 18:11:30 +00:00
orbiter
75d1437340 fix for http://forum.yacy-websuche.de/viewtopic.php?p=1123#p1123 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4002 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2007-07-24 13:39:53 +00:00
orbiter
9ca46a8c69 indexing of local (intranet) urls enabled 
To do this, one must create a separate YaCy network that has a local URL domain
A description how to do this is here: http://www.yacy-websuche.de/wiki/index.php/De:Netzdefinition

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4001 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2007-07-24 00:46:17 +00:00
orbiter
c48c9735cb bugfix for property table loader (did not exclude comment lines) 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4000 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2007-07-23 13:38:17 +00:00
orbiter
6758beae9c fix for http://forum.yacy-websuche.de/viewtopic.php?p=1092#p1092 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3999 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2007-07-23 09:54:12 +00:00
orbiter
511dcbb172 fixed encoding bug made in SVN 3993 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3998 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2007-07-23 00:50:57 +00:00
orbiter
ec817a2ff5 removed JAR command from ftpc - produces warnings and is not used by YaCy 
see http://forum.yacy-websuche.de/viewtopic.php?f=5&t=193&hilit=

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3997 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2007-07-22 20:08:09 +00:00
orbiter
f5a4efb76e fix for http://forum.yacy-websuche.de/viewtopic.php?f=6&t=192&hilit=&p=1034#p1034 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3996 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2007-07-20 08:06:21 +00:00
orbiter
bf6f6820ec fix for another BMP parser problem. This is not a bug, I guess it was caused by a corrupted graphics file. 
see http://forum.yacy-websuche.de/viewtopic.php?f=6&t=180&hilit=&p=1017#p1017

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3995 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2007-07-19 22:11:48 +00:00
orbiter
40b0547611 - documentaton changes (removed old forum links) 
- different handling of link quotation
- different handling of link normalization
- enhanced html/unicode en/de-coding

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3993 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2007-07-19 15:32:10 +00:00
orbiter
dcb8687904 fix to update cycle 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3992 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2007-07-18 22:45:29 +00:00
orbiter
cff019464e fix for BMP/ICO magic detection 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3991 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2007-07-18 13:08:24 +00:00
orbiter
557f8d80e4 - better logging 
- fixed bugs in auto-update

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3990 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2007-07-18 13:03:20 +00:00
orbiter
f323e1813d added commons.logging again (is used by mimeTypeParser) 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3989 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2007-07-18 11:47:19 +00:00
michitux
a695c93662 Fix of the Status-page for IE: 
- reverted revision 3982 and 3979
- added a real fix for http://forum.yacy-websuche.de/viewtopic.php?f=6&t=163


git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3988 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2007-07-17 22:06:06 +00:00
orbiter
b6d9cca67e - fixed problem with yacyVersion and own version generation 
- within this context: generalized date format handling
- extended Update interface:
 * a version lookup can be triggered manually
 * a complete lookup + download + re-boot process can be triggered with one click

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3986 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2007-07-16 23:47:21 +00:00
orbiter
b1630e73c8 addition to fix 3983 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3985 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2007-07-16 14:43:40 +00:00
orbiter
6071668c3b better error message in case that a mime type cannot be found. 
see also http://forum.yacy-websuche.de/viewtopic.php?f=6&t=132&p=587#p587

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3984 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2007-07-16 14:40:19 +00:00
orbiter
d477773fa3 fix for b/w color mode in BMP parserhttp://forum.yacy-websuche.de/viewtopic.php?f=6&t=180&hilit=&p=942#p942 
see also 

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3983 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2007-07-16 14:27:37 +00:00
orbiter
e7a99d765e fix for http://forum.yacy-websuche.de/viewtopic.php?f=6&t=171 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3977 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2007-07-16 12:53:14 +00:00
orbiter
03847bebc1 removed unused libs 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3971 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2007-07-16 10:34:39 +00:00
orbiter
9da0e53fe8 repaired rss feed reader 
- removed old rss parser
- removed unused rss parser libraries
- added new rss reader
- added previously removed FeedReader_p.java and adopted it to new rss parser
- adopted parser interface for rss indexing to new rss parser

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3970 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2007-07-16 10:07:48 +00:00
orbiter
26ddf797eb added bmp and ico image format to all parser/viewing methods 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3969 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2007-07-15 12:55:41 +00:00
orbiter
5444b07674 fixed bug with decompression of index abstracts 
this fixes a problem that occurred when searching for several words

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3968 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2007-07-15 12:39:16 +00:00
orbiter
924ae39170 replaced old map loading method with new implementation which is more robust against change of line termination methods 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3967 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2007-07-15 11:45:41 +00:00
orbiter
ac7b6b9c10 added a default favicon for search results from domains that do not have a favicon 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3966 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2007-07-15 01:37:43 +00:00
orbiter
89e1848db6 fixed problem with favicons: 
target servers had been able to see search words from the referrer of the favicon fetch.
This has been removed by using the getImage - servlet for favicon fetch.
Since java does not support loading of bmp and ico-Images, such parsers had been added.
The image parser had been coded from their original microsoft documentation.
This influences also the image-search functionality: there can now be a preview
of found bmp-images. Another benefit: favicons for search results are now cached with the HTCACHE.

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3965 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2007-07-15 01:34:01 +00:00
low012
d95a06e2e4 *) cosmetics: fixed a typo 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3964 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2007-07-14 12:49:11 +00:00
orbiter
7c5c814a47 - simplified code (removed exception handling where not necessary) 
- added confirmation dialog for shutdown and restart

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3962 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2007-07-13 14:54:01 +00:00
orbiter
a4e8ad95ab enhancements to news and switchboard queue processing 
removed direct access and replaced by iteration

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3961 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2007-07-13 13:00:18 +00:00
orbiter
a45216b479 fix to prevent bad-formed news messages 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3960 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2007-07-13 09:41:55 +00:00
orbiter
bec4dbc753 added options and execution methods for automated updates 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3959 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2007-07-12 16:23:33 +00:00
orbiter
527b3decde - re-sructuring of configuration menus 
- added new system update configuration page
- moved system update from status page to system udate page
- moved shutdown and restart from status page to main menu
- added new configuration properties to yacy.init (not yet actively used)
- added some methods to handle new automatic update process

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3958 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2007-07-10 23:56:25 +00:00
orbiter
4968556668 - fix for broken news queue during iteration 
- enhancement for searching special news (usage of new iterator)

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3957 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2007-07-10 21:22:06 +00:00
low012
1ea5fa2c04 *) Changed a comment to get rid of this message: 
[javac] /home/low012/subversion/yacy/trunk/source/de/anomic/http/httpc.java:1117: warning: unmappable character for encoding UTF8
    [javac]             // if download == null, the get result is stored to a byte[]�and returned,
*) Changed broken link (see: http://forum.yacy-websuche.de/viewtopic.php?t=128)


git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3956 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2007-07-07 09:04:41 +00:00
orbiter
5702257a5f fix for elementCount bug when db was reset 
possible solution for http://forum.yacy-websuche.de/viewtopic.php?f=5&t=124&hilit=

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3955 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2007-07-06 09:50:59 +00:00
orbiter
208b5297f1 enhanced handling of news records: 
result is a speedup of Surftips, Supporter, and Network page

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3954 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2007-07-05 22:56:37 +00:00
orbiter
f40566f9bb separate YaCy networks: 
- added server-side network unit identification
- added server-side network access authorization
- enhanced client-side network authentification essentials generation
- implemented first peer-peer salted-magic authentification method

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3953 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2007-07-04 23:48:52 +00:00
orbiter
a9e73b6852 fixed great mess with localization paths. the problem was: 
automatic re-translation after update did not work. hopefully now

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3952 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2007-07-04 10:32:30 +00:00
orbiter
53266e6497 does not belong to yacy, just a test file (committed by mistake) 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3951 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2007-07-04 09:58:22 +00:00
orbiter
36a37f758b fix for oom exception during release download 
see http://forum.yacy-websuche.de/viewtopic.php?f=6&t=101&hilit=

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3950 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2007-07-03 22:55:47 +00:00
orbiter
3421c64d26 implemented update function: 
after downloading a release using the download button on the status page
the user can choose any of the downloaded versions for a update.
this enables also a downgrade to a older version.
when the update button is pushed, yacy terminates, installes the choosen version
and restarts

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3948 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2007-07-02 15:16:05 +00:00
orbiter
21fabe259b another fix to the restart function; now tested under linux 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3947 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2007-07-01 22:43:08 +00:00
low012
2158f83d43 *) cosmetics, changed a character to get rid of "warning: unmappable character for encoding UTF8" during compilation 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3946 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2007-07-01 17:17:29 +00:00
orbiter
28baecd41b another fix for the concurrentModificationException in AccessTracker 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3944 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2007-06-30 23:01:57 +00:00
orbiter
84be912e90 fix for null pointer exception that occurred when missing user-agent in request header 
see also http://forum.yacy-websuche.de/viewtopic.php?f=6&t=78&hilit=

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3943 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2007-06-30 18:08:10 +00:00
orbiter
c1aad9e508 added parameter for network graphic background 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3942 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2007-06-29 23:06:13 +00:00
orbiter
19786b73b6 next try for a better restart 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3941 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2007-06-29 21:36:18 +00:00
orbiter
08d5db6bb4 next try to fix the restart 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3939 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2007-06-29 14:45:53 +00:00
orbiter
2ff94b2fb4 another try to fix the restart on linux (it works on mac) 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3938 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2007-06-29 13:56:28 +00:00
orbiter
c5c268c43e tried to fix restart button 
** kann das mal jemand auf seiner linux-platform testen **
** und feed-back geben ob der restart funktionier ? **

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3937 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2007-06-29 12:46:08 +00:00
orbiter
e03fcf4627 SSI fix for http://forum.yacy-websuche.de/viewtopic.php?f=6&t=29 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3936 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2007-06-29 10:45:13 +00:00
orbiter
1a45ecb356 - fix for http://forum.yacy-websuche.de/viewtopic.php?f=5&t=14&p=137#p137 
- fix for missing restart script in ant built target
- removed some more synchronization for size() operations
- removed blocking statement on search page

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3935 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2007-06-28 22:06:33 +00:00
orbiter
1fa4feb8e6 added restart button. should work on linux and mac, but was only tested on mac 
should of course work on windows as before

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3934 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2007-06-28 20:36:15 +00:00
orbiter
f1ed91a8e4 added option to allow/disallow DHT transmission during indexing 
see also http://forum.yacy.de/viewtopic.php?f=9&t=8

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3933 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2007-06-28 15:25:33 +00:00
orbiter
87afdfc2a7 fix for long waiting time during deletion of processed news 
see http://forum.yacy.de/viewtopic.php?f=6&t=6

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3932 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2007-06-28 15:03:58 +00:00
orbiter
9bbd39b67c - removed unfinished auto-updater from roland and martin 
- added new download-option for releases on the status page
still mising:
- thomas-style restart for linux/mac
- untar/gunzip on shell basis
(comes next)

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3931 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2007-06-28 14:52:26 +00:00
orbiter
154ffd7c2c fix for wrong http connection version and SSIs 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3928 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2007-06-26 15:06:23 +00:00
orbiter
1782ef57e5 - added SSI parser and include directive for <!--# include virtual="<file>" --> 
- added chunked file transfer for non-yacy clients
- SSIs are streamed using chunked transfer, partly delivered pages can be seen in browser before transmission is finished
- added client-side network unit identification
- cleaned up code

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3926 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2007-06-26 14:37:10 +00:00
orbiter
6b4cfbd2d6 new network bootsraping method 
- no more contact to yacy.net (no remote superseed any more)
- moved superseed file into new network unit definition
- fixed build; includes new network bootstraping files now

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3922 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2007-06-25 14:43:57 +00:00
orbiter
0e57a8062b added network definition for different YaCy networks 
(needs much more work)

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3919 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2007-06-22 14:29:14 +00:00
auron_x
1d41ebf489 *) made age for deletion of too old seeds configurable 
*) changed naming-scheme of seed-deletion-properties

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3918 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2007-06-22 10:42:29 +00:00
auron_x
52cb3208d0 *) old (lastseen > 7d) peers are now automatically removed from passive and potential seed-dbs 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3917 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2007-06-22 09:16:25 +00:00
orbiter
815e3da62f fix for http://www.yacy-forum.de/viewtopic.php?p=37353#37353 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3913 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2007-06-18 11:34:30 +00:00
michitux
25529290ca - 2 small changes in documentation 
- hopefully fixed logging of GCs (in order to avoid things like "performed necessary GC, freed 18014398509481565 KB (requested/available/average: 4096 / 1631 / 2957 KB)") with the help of KoH


git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3909 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2007-06-17 19:32:38 +00:00
low012
c59a7ce5c2 *) hopefully fixed a stupid bug (my fault of course) that sometimes messed up the marking of search words in the snippets (see http://www.yacy-forum.de/viewtopic.php?p=37329#37329) 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3908 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2007-06-17 16:29:04 +00:00
orbiter
6518bb6c08 changed release strategy: 
we will provide two different releases in the future, one standard release and one 'pro'-release.
the 'pro'-release contains all additional parsers AND has different default performance values.
The pro-version differs therefore from the previous 'all'-version by this default values.
The pro-configuration is automatically choosen if the libx-folder exists. If a version is once initialized, its configuration stays independently from an existing libx folder.
The ant targets had been changed. There are now 3 different targets to create standard and pro-releases, and one target to upgrade:
- dist: creates a standard release (only, no libx target any more)
- distPro: creates a pro-release (includes the libx)
- distExt: creates a libx-release which includes the libx-folder only. It may be used to upgrade from standard to pro
Furthermore, the naming of 'dev'-releases had been removed.

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3902 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2007-06-16 14:11:52 +00:00
orbiter
069562a14d fixed problem with re-crawl; replaced error file-db with ram-db 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3900 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2007-06-15 23:47:08 +00:00
orbiter
c7a614830a several bugfixes 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3899 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2007-06-15 17:45:49 +00:00
allo
465145cb6f revert to insecure, but dau-proof defaults 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3898 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2007-06-15 12:56:52 +00:00
allo
7ad11ceaaa security fix for peers without password. allow access only from localhost 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3897 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2007-06-15 00:03:44 +00:00
(no author)
2784820ee3 *) moving sleep to a better place 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3895 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2007-06-14 16:53:22 +00:00
theli
7a1b811d18 *) bugfix for SocketException: 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3893 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2007-06-14 15:58:10 +00:00
orbiter
71fd972ac0 - reduced default search time 
- catched case when web structure cannot be painted because of too less data
- better logging when balance fails


git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3892 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2007-06-14 15:21:01 +00:00
orbiter
2b937abef1 slighlty different behavior in shutdown sequence for http server threads: 
- first close streams
- make pause (that one that was made in httpdFileHandler)
- close sockets

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3890 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2007-06-14 11:58:20 +00:00
orbiter
e4aa8f2a08 disabled more sleep(200) 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3889 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2007-06-14 11:02:28 +00:00
orbiter
cb38e57622 reduced httpd final waiting time 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3888 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2007-06-14 10:28:37 +00:00
orbiter
b4585ad67d im Sommer 2005 wurden die ersten pings zwischen YaCy-Peers ausgetauscht. 
Das klappte aber merkwürdigerweise nicht immer. Um das Protokoll zu testen schrieb ich eine einfache message-Funktion, so wie sie heute noch in YaCy drin ist.
Aber auch die Messages funktionierten nicht richtig. Alex und ich haben lange Zeit gesucht, und den Fehler nie gefunden. Es stellte sich heraus das ein Timing-Detail das Problem lösen konnte, die Ursache haben wir bis heute nicht gefunden.
Die Lösung des Problems bestand aus einem kurzen sleep, kurz bevor der httpd Daten zum client zurück geschrieben hat. Das ist natürlich eine fürchterlich schlechte Lösung.
Bis heute war diese Sache im httpd. Mit diesem Commit habe ich den sleep auskommentiert, und es steht zu befürchten das wieder irgendwas nicht geht.
Wenn jetzt das Netz zusammenbricht, keine pings mehr ankommen oder so, war es dieses sleep, das es verhinderte.
Vorschläge willkommen.

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3887 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2007-06-14 08:18:10 +00:00
orbiter
f64d9f1c6c removed forced termination in case that a previous bad termination is detected 
this will cause many users to be unsure what to do next an leave them helpless
to simply delete the control file is the same thinig that the user is othervise forced to do

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3885 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2007-06-13 20:55:48 +00:00
karlchenofhell
e1d809d5f1 - more detailed logging of MEMORY messages 
- forced GCs don't contribute to heuristics anymore

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3881 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2007-06-13 15:03:56 +00:00
orbiter
9f7765863b bugfix for seed length control routine 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3879 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2007-06-13 13:21:19 +00:00
orbiter
0b10ef64ba better server access tracking 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3878 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2007-06-13 13:05:51 +00:00
orbiter
4f5496062c protection against too large seeds 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3877 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2007-06-12 22:08:33 +00:00
orbiter
684ded0e09 added new news types 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3876 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2007-06-12 15:15:24 +00:00
karlchenofhell
669f840eab - added ViewProfile / Impressum (default on) to local peer's robots.txt 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3874 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2007-06-12 13:48:27 +00:00
orbiter
d7de0938a6 fix for http://www.yacy-forum.de/viewtopic.php?p=36587#36587 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3870 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2007-06-12 08:47:34 +00:00
karlchenofhell
22ee85ca02 - specified exceptions thrown by ResourceInfoFactory and plasmaHTCache.loadResourceInfo() 
- caught possible NPE in CacheAdmin_p and added more error-cases
- speeded up deletion of entries in the local crawl queue by crawl profile (it has been noted often that this deletion is slow)
- added a bit javadoc

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3868 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2007-06-11 23:33:24 +00:00
orbiter
dfd5e823c3 automatic limitation of web structure host count 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3867 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2007-06-11 22:08:10 +00:00
orbiter
8b0aea6910 fixed automatic deletion of too many referenced hosts in web structure 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3866 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2007-06-11 21:51:56 +00:00
orbiter
5dd9acc2a7 removed calls to deprecated methods 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3865 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2007-06-11 21:33:45 +00:00
orbiter
66ec8b63c1 added a httpd access tracker: 
- all requests to the own httdp can now be listed in the access tracker menu
- the search statistics had been renamed to access tracker and extended by this tracker

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3861 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2007-06-11 14:05:20 +00:00
karlchenofhell
71ca9aa6d4 - fix for changed blacklist types 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3857 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2007-06-11 09:56:16 +00:00
orbiter
9a8a87612d added new qph column to search tracker servlet 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3854 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2007-06-10 22:02:17 +00:00
orbiter
e07458bad4 added time-out function to web analysis 
the default time-out is 1 second

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3852 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2007-06-10 20:00:44 +00:00
hydrox
4a1bc4743a *)News-entries with blacklisted URLs are now ignored 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3849 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2007-06-10 08:05:18 +00:00
allo
6074264267 dynamic rights. 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3847 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2007-06-09 19:34:09 +00:00
theli
99062c0c9e *) SOAP should support authentication against the user-DB now (requested by KoH) 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3846 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2007-06-09 15:49:04 +00:00
theli
339153d40e *) favicons that are specified in the document content via html link-tags 
are now detected and displayed on the search page (requested by allo).

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3845 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2007-06-09 15:22:37 +00:00
allo
854eb1492f .yacy /.yacyh urls for the feedreader 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3844 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2007-06-09 12:56:08 +00:00
allo
7a5b22a0b8 Integration of FeedReader in Bookmarks. 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3841 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2007-06-08 23:27:42 +00:00
karlchenofhell
6265d321bd - more constants 
- display why global search is not available on search page

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3839 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2007-06-08 20:01:16 +00:00
allo
7921f07c9d userDB fix 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3837 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2007-06-08 16:11:10 +00:00
allo
7b2e1bb8f2 Feedparser with reflection. 
TODO: This needs a special build.xml entry


git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3832 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2007-06-08 14:31:09 +00:00
karlchenofhell
8bff810d19 - fixed logging output of serverMemory.request() 
- don't start up if DATA/yacy.running exists as this is usually a sign of an already started yacy-instance

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3831 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2007-06-08 12:45:03 +00:00
rramthun
18a5380ee3 *) situation-dependent lock-buttons for search-page 
*) removed one unused import and a double definition of "ogg" as media-type

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3817 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2007-06-07 15:26:41 +00:00
karlchenofhell
f05ca43780 - the wiki-parser works for remote wiki-code now, not displaying links anymore as if they were local (ViewProfile comment) 
- fixed wrong link to CrawlStart on Status-page

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3816 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2007-06-07 11:35:48 +00:00
karlchenofhell
9d6605a83c - fixed NPE in Blacklist Cleaner during deletion of more than one double entries 
- don't display responseHeader1.db in CacheAdmin_p anymore

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3814 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2007-06-06 23:36:38 +00:00
orbiter
594ff95955 :-( 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3801 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2007-06-06 11:34:39 +00:00
orbiter
4ca797401e fix for ConcurrentModificationException 
see http://www.yacy-forum.de/viewtopic.php?p=36566#36566

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3800 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2007-06-06 10:36:04 +00:00
orbiter
7b904e0077 integrated robots.txt crawlDelay into the crawl balancer 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3797 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2007-06-06 07:53:56 +00:00
orbiter
52cb033f01 - slightly different painting of web structure picture: 
hosts that have many own connections are painted farer away (this is not yet cato's idea, this will be implemented in another step)

- doc update

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3796 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2007-06-05 15:32:43 +00:00
karlchenofhell
30c3d909b1 - fixed charset problem in ConfigProfil_p.html (use accept-charset="UTF-8" in forms) 
- fixed wrong XML output if no peers are known in Network.xml
- simplified parsing of table properties in wikiCode and ZTableToken
- reimplemented GC heuristics. They are needed to constantly ensure that an amount of free memory is available which is higher than Java's max. limit for performing a Full GC (please use serverMemory.request(long, boolean) rather than serverMemory.available(long, boolean) to provide data for averaging over the last GCs)

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3793 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2007-06-05 11:37:19 +00:00
allo
6c9df13552 more debugging 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3791 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2007-06-04 20:30:40 +00:00
allo
4392ee0c51 BugFix for typo and wrong include 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3789 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2007-06-04 16:06:58 +00:00
allo
d1e1580223 Surftips Blacklist 
Blacklists List Hardcoded instead of only updated on firststart / migration.java

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3788 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2007-06-04 15:36:10 +00:00
theli
e1a5babff1 *) Logging GUI handler: line-size is now set to max-size if max-size was exceeded 
See: http://www.yacy-forum.de/viewtopic.php?p=36355

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3786 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2007-06-02 21:23:32 +00:00
(no author)
94cc9f05f5 *) Improvements for restart via update wrapper 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3785 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2007-06-02 15:25:13 +00:00
hydrox
44bac7dea1 *) blog-comments can now be moderated 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3778 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2007-06-01 06:02:55 +00:00
borg-0300
2ab020445a bugfix, i think - http://www.yacy-forum.de/viewtopic.php?t=4059 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3777 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2007-05-31 17:03:02 +00:00
(no author)
f89517203d *) SOAP: new function to get the Performance Settings of Queues and Processes 
No items left in the yadmin SOAP-TODO :-)

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3776 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2007-05-31 15:47:58 +00:00
(no author)
92351c4dcb *) SOAP: bookmarks list now indicates if a bookmark is private (requested by KoH) 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3775 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2007-05-31 14:54:56 +00:00
allo
957a25afff getRight(rightName) instead of get...Right() 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3774 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2007-05-31 14:48:20 +00:00
(no author)
e0e46d3aec *) SOAP: new function doGarbageCollection (requested by KoH) 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3773 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2007-05-31 14:32:41 +00:00
(no author)
1efe607c34 git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3771 6c8d7289-2bf4-0310-a012-ef5d649a1542 2007-05-31 11:24:59 +00:00
low012
a0149317ac *) fixed bug where headlines were added to directory of a wiki page multiple times (http://www.yacy-forum.de/viewtopic.php?t=4034) 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3762 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2007-05-25 16:36:09 +00:00
(no author)
ef24bed406 Sorry... 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3760 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2007-05-24 16:25:07 +00:00
(no author)
a29cb2e1af blupp 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3759 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2007-05-24 16:14:46 +00:00
theli
651b05ba43 *) wsdl file updated (requested by KoH) 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3758 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2007-05-24 16:07:58 +00:00
orbiter
0ae6664ad8 enhanced web structure picture 
- hand-over of get properties from web front-end to graphics generation
- added depth-control buttons
- added marking of anchor-points to highlight relation order
- enhanced ymage graphics library

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3757 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2007-05-24 15:28:03 +00:00
orbiter
a585b4d41b added web structure image 
see http://localhost:8080/WatchWebStructure_p.html

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3747 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2007-05-22 15:20:50 +00:00
orbiter
33ad0c8246 added a web structure computation and logging: 
- all web page parsing operations will now increase a web structure file
- the file is computed in memory and dumped at shutdown-time to PLASMASB/webStructure.map in readable form (not a database)
- the file can be used externally to analyse the link structure of the crawled pages
- the web structure can also be retrieved using a xml-interface at http://localhost:8080/xml/webstructure.xml
- the short-term purpose is the computation of a link-graph image (before linuxtag!)
- a long-term purpose could be a decentralized computation of the citation rank



git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3746 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2007-05-22 08:13:48 +00:00
karlchenofhell
7904175338 - sorry for typos 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3743 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2007-05-20 16:22:46 +00:00
karlchenofhell
baa9402b97 - wiki-parser is now configurable via the config setting wikiParser.class which holds the class-name for the parser to use 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3742 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2007-05-20 16:19:25 +00:00
karlchenofhell
601fc7d1c5 - added source to J7Zip-modifed.jar and it's license (changelog is still to come) 
- moved HTML-*replace-methods from wikiCode to de.anomic.data.htmlTools
- prepared use of different wiki parsers as suggested here: http://www.yacy-forum.de/viewtopic.php?p=34444#34444

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3741 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2007-05-20 13:29:12 +00:00
karlchenofhell
0a64047081 - plasmaParserDocument can process subdocuments now (other archive-parsers may want to use this method) 
- added 7zip parser
- added 'text/sgml' to realtime parseable mimetypes (sometimes returned by the mime type parser)
- added new cached output stream class, very suitable for parsers because of limited memory

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3740 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2007-05-18 23:13:44 +00:00
theli
b1680ab71f *) bugfix for ArrayIndexOutOfBoundsException in robots-parser (thanks to low012) 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3739 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2007-05-18 13:39:08 +00:00
theli
b30e64daab *) passing homepath to serverLog.configureLogging 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3738 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2007-05-18 13:04:26 +00:00
theli
9a4375b115 *) robots.txt: adding support for crawl-delay 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3737 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2007-05-18 13:00:42 +00:00
orbiter
11ac7688d5 reverted a part of last commit 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3736 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2007-05-16 17:52:11 +00:00
orbiter
b3f97b5c38 git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3735 6c8d7289-2bf4-0310-a012-ef5d649a1542 2007-05-16 17:45:39 +00:00
orbiter
3c5ff7f735 adopted kelondroBytesIntMap to kelondroIntBytesMap 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3734 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2007-05-16 15:20:15 +00:00
orbiter
5551ff5306 enhanced index storage data structure kelondroBytesIntMap 
this stores now two index structures, one for data that is aquired during start-up
and one for data that is aquired during run-time. This reduces the grow factor, and should reduce the memory amount in case that a index-reorganisation happens.

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3733 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2007-05-16 14:36:56 +00:00
orbiter
872eb46cb9 some redesign of the handling of the index for kelondroFlexTable 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3732 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2007-05-16 10:48:26 +00:00
karlchenofhell
086239da36 - added servlet: remote crawler queue overview 
- added servlet: crawl profile editor

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3731 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2007-05-16 10:11:25 +00:00
karlchenofhell
8ba81e0995 - added some comments (will get more in the near future) 
- added missing <label> to the search field in Network.html

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3728 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2007-05-15 15:48:28 +00:00
allo
65a8a9fc58 fix for nullpointer 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3726 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2007-05-14 16:56:13 +00:00
orbiter
b05e2314cf another dht selection fix 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3725 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2007-05-14 12:52:39 +00:00
karlchenofhell
03c6551b0c - fix for http://www.yacy-forum.de/viewtopic.php?t=3747 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3724 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2007-05-14 12:44:44 +00:00
orbiter
b28e5d0ee9 protection against wrong word hash length 
see http://www.yacy-forum.de/viewtopic.php?p=35657#35657

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3723 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2007-05-14 10:00:23 +00:00
orbiter
e897eb9b4a fix for DHT selection target 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3722 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2007-05-14 08:17:36 +00:00
orbiter
0384b8771b fix for http://www.yacy-forum.de/viewtopic.php?p=35700#35700 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3719 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2007-05-13 19:37:16 +00:00
orbiter
578c2ef130 release 0.52 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3715 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2007-05-11 22:12:29 +00:00
orbiter
756a27049b last-minute-feature 'newbie-selection' for workshop purpose: 
for remote search, always select all peers that are less than a day old
(should be removed someday in the future if load is too high, which could mean when pph > 100)

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3712 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2007-05-11 19:55:19 +00:00
orbiter
46367afaaa update of memory-protection values 
see http://www.yacy-forum.de/viewtopic.php?p=35539#35539

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3709 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2007-05-11 18:02:48 +00:00
rramthun
ea87fe5d78 *) Updated German translation 
*) Changed "Lost Handle" error to warning (masses of it if deleting crawl-profile)
*) Removed unnecessary code from Windows script 

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3708 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2007-05-11 17:48:22 +00:00
orbiter
85035dc319 addition to svn 3699: check send/receive if p2p-mode is activated 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3701 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2007-05-10 13:27:38 +00:00
orbiter
26f05d1fd0 avoid division by zero if search is done for no words 
this case is relevant if the bluewords (yacy.blue) are used

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3698 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2007-05-09 22:10:12 +00:00
orbiter
2fa8b50e54 reverting svn 3691+3692 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3696 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2007-05-09 19:31:40 +00:00
orbiter
139c59ebbd - fixed dht selction problem: the seed tables used a wrong ordering 
- cleaned some code

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3693 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2007-05-09 17:59:36 +00:00
orbiter
22a0e9f117 more timeout-control 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3692 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2007-05-09 14:53:17 +00:00
orbiter
24db55a541 added timeout for httpd-sockets during read 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3691 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2007-05-09 14:30:01 +00:00
orbiter
f3fee4d445 fix for http://www.yacy-forum.de/viewtopic.php?p=35322 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3689 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2007-05-08 09:19:36 +00:00
orbiter
7f56c8d4aa fixed some seed selection details 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3685 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2007-05-07 22:22:35 +00:00
orbiter
e602436fda fixed problem with cluster routing 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3684 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2007-05-07 20:48:24 +00:00
orbiter
0831034e07 fixed non-termination bug for robinson remote crawl peer selection 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3681 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2007-05-07 14:37:50 +00:00
orbiter
68f5d64ae6 replaced yacy logo by better version 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3675 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2007-05-07 08:54:27 +00:00
orbiter
d6480dc670 fix for long transfer pauses 
see http://www.yacy-forum.de/viewtopic.php?p=35243#35243

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3672 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2007-05-06 21:43:20 +00:00
theli
cb43ae11ba *) Bugfix 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3668 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2007-05-06 12:57:22 +00:00
theli
0b5fc3c28c *) moving date functions to serverDate class 
*) Sitemap-parser
   - logging added
   - parsing of modDate added

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3667 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2007-05-06 12:36:49 +00:00
theli
6f46245a51 *) Bookmarks: Ajax icon is displayed while loading title 
*) First version of a sitemap parser added
   - currently only autodetection of sitemap files is supported
*) DB-Import restructured
   - pause/resume should work again now


git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3666 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2007-05-06 09:52:04 +00:00
theli
269d5ca45b *) Bugfix for IllegalMonitorException 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3664 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2007-05-06 08:54:02 +00:00
theli
e8610f049a *) new method allowing the updater to wait until yacy has finished startup 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3662 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2007-05-06 08:22:18 +00:00
orbiter
111ba9e359 - fixed some width problems in new status page 
- fixed deadlock in dns cache
- added termination security for DHT peer selection

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3660 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2007-05-05 23:18:00 +00:00
theli
74dd6cac95 *) signal yacy shutdown to updater 
*) some javadoc added

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3658 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2007-05-05 16:20:01 +00:00
theli
43748f87fb *) changes required for the uploader 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3655 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2007-05-05 15:41:05 +00:00
rramthun
e12e934ade *) Fixed broken compile process. 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3650 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2007-05-04 21:33:37 +00:00
rramthun
d6811ac243 *) Moving tar.jar from libx to lib 
*) Enhanced interface

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3649 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2007-05-04 19:46:23 +00:00
theli
469583ea80 *) new interface class. should be implemented by the updater to allow communication between the updater and yacy 
(not yet functional)

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3648 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2007-05-04 14:22:10 +00:00
orbiter
191ef16499 fixed wrong ordering that caused bad dht selection 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3646 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2007-05-04 14:11:50 +00:00
orbiter
7cf8981a98 - added debugging code for wrong DHT target iterator 
- restricted distance constraint from 0.4 to 0.2

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3644 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2007-05-03 22:57:55 +00:00
rramthun
abb63e3289 *) disabled system.exit() in case of YaCy shutdown as it kills the whole VM (including updater and other management threads) 
*) UpdateCheck-Thread now pauses for given interval correctly

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3642 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2007-05-03 19:16:43 +00:00
orbiter
dd44a1394f disabled automatic performance setting change 
- during crawl start
- each indexing cycle
- for delay values
- for short memory cycles

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3634 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2007-05-02 15:39:27 +00:00
orbiter
b9add5cf37 some bugfixes: 
- dht iterator start point
- wordIndex synchronization
- surftipps url check

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3633 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2007-05-02 14:20:43 +00:00
orbiter
06b6e35484 fix for a null pointer exception if clusters are not defined 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3632 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2007-05-02 12:26:29 +00:00
orbiter
35c660654d more debugging lines to fix bug for 
http://www.yacy-forum.de/viewtopic.php?p=34935#34935

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3629 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2007-04-30 23:05:19 +00:00
orbiter
47e90f31b2 fix for deadlock in plasmaWordIndex.addPageIndex 
synchronization for class method not necessary
see also: http://www.yacy-forum.de/viewtopic.php?p=34959#34959

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3628 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2007-04-30 22:30:09 +00:00
orbiter
81844e85b2 - fixed more cluster routing problems 
- fixed a problem in remote search when balancer caused shift process to wait too long

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3627 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2007-04-30 00:39:53 +00:00
orbiter
304ed3f4d2 fix for remote crawl requests in clusters 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3626 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2007-04-29 22:52:07 +00:00
orbiter
1979a167d3 fixed problem with cast 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3625 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2007-04-29 22:31:44 +00:00
orbiter
e48189c710 enhanced cluster routing 
- cluster definitions can now contain an addition for local ip addresses
- cluster-cluster communication uses the local ip address instead the global address, if one is given

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3624 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2007-04-29 22:05:34 +00:00
theli
e9d87b2fce *) changes required for the uploader 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3621 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2007-04-29 15:56:45 +00:00
orbiter
b33cef421e better routing for public clusters 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3620 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2007-04-29 00:08:38 +00:00
theli
7c902996b5 *) changes required for the uploaderWrapper 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3618 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2007-04-28 16:04:37 +00:00
orbiter
f73e1e3af9 fixed bugs in remote search setting for public clusters 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3615 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2007-04-28 00:12:41 +00:00
orbiter
485bf1ea83 bugfix for robinson/remote crawl bug 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3614 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2007-04-27 21:35:43 +00:00
orbiter
62c947b4aa next try to fix deadlock in plasmaWordIndex 
see also:
http://www.yacy-forum.de/viewtopic.php?p=34821#34821

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3607 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2007-04-27 12:26:36 +00:00
orbiter
871ee1ce0f one step closer to automatic updates: 
automatically aquire release information from download archives
web pages from latest.yacy-forum.net and yacy.net are retrieved, parsed,
links wihin are analysed, sorted and the most recent developer and main
releases are provided as direct download link on the status page, if it was
discovered that a more recent version than the current version is available.
This process is done only once during run-time of a peer, to protect our
download archives from DoS by YaCy peers.

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3606 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2007-04-27 09:23:44 +00:00
theli
ec225f9ab6 *) SOAP: adding methods to get the comment and MD5 checksum of a single file 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3604 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2007-04-27 05:10:31 +00:00
theli
2399ed817c *) robots.txt parser now extracts the sitemap-URL (will be used later) 
*) some javadoc added
*) junit testclass for robots.txt parser added

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3602 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2007-04-26 15:42:38 +00:00
orbiter
fa012789b2 tried to fix a deadlock problem durin shutdown 
see also:
http://www.yacy-forum.de/viewtopic.php?p=34753#34753

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3601 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2007-04-26 15:15:40 +00:00
orbiter
e192f616a2 collection of small bugfixes 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3600 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2007-04-26 14:28:57 +00:00
orbiter
64a6d6e5e6 added new set iterator (needed for last commit) 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3599 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2007-04-26 09:52:37 +00:00
orbiter
f8de19fb2f robinson cluster: added client-side protocol implementation 
- the network configuration page shows a new option: robinson clusters
- when a global search is made, all robinson peers are excluded, but:
- robinson peers/clusters that provide peer tags and where search words match
  such tags, they are included in global search. Therefore, robinson peers/clusters
  support the global yacy network with their indexes, without doin DHT-exchange


git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3598 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2007-04-26 09:51:51 +00:00
theli
50e1e61fa5 *) SOAP: adding functions to rename and move files 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3595 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2007-04-25 05:36:28 +00:00
theli
63a004abff *) bugfix for Nullpointerexception 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3594 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2007-04-25 05:22:09 +00:00
orbiter
29fe2beac7 possibly fixed a deadlock 
cannot find forum link now for that

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3593 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2007-04-24 21:55:57 +00:00
(no author)
4f4d3d71dd *) Faster appearance of ConfigBasic by bypassing UPNP-scan in case of existing external connects 
*) Marked two deprecated source-points
*) Added possibility to dump words from indexing to file. Should not affect performance in the current form.

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3592 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2007-04-24 16:33:31 +00:00
orbiter
657585fe0d network functions for robinson peers: server-side protection 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3591 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2007-04-24 15:11:12 +00:00
orbiter
89c1511738 - added new Network Configuration menu, can be found in basic settings 
- new cluster functions will be available in this menu, but currently not enabled,
  because corresponding interface methods are not ready yet
- shifted remote crawl settings to new network configuration menu
- shifted DHT distribution/receive to the new network configuration menu
- adopted some string constants
- added cluster configuration settings to yacy.init


git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3589 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2007-04-23 20:47:07 +00:00
orbiter
62b79aa0a9 bugfix for http://www.yacy-forum.de/viewtopic.php?p=34558#34558 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3586 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2007-04-20 21:38:12 +00:00
orbiter
2f3b518169 temporary patch for startup-problem: 
http://www.yacy-forum.de/viewtopic.php?t=3854
This is a serious problem that is caused by the database bug between 0.511 - 0.513
which produced a large number of double-entries in the RWI index. The uniq()-method
tries to fix this, and it does not terminate when the index is large and the number
of double-occurrences is also large. This patch does simply implement a time-controlled
termination, which does not heal the inconsistency problem. The uniq-method itself
is correct and does not need a bugfix, the non-termination is simply caused by the large number
of data that is shifted during the process. It was possible to reproduce this behaviour
in a test environment.
A real fix would need to:
- enhance the uniq()-method by using a recursive, binary segmentation of the array to be fixed
- uniq() must report the entries that are double
- the double-entries must be deleted from the collection index (from the index and the collections) to heal the problem


git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3583 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2007-04-20 07:53:58 +00:00
rramthun
e6fb6426a3 *) Some cosmetical changes and corrections 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3582 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2007-04-19 16:16:54 +00:00
orbiter
ba525ebf52 - re-enabled path optimization that was disabled during testing 
- re-implemented index load/extend optimization that was removed from kelondroFlexTable,
  this is now part of kelondroIntBytesIndex


git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3580 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2007-04-19 14:55:19 +00:00
orbiter
595ee10468 fixed datatabase inconsistency bugs 
inserted many debug lines
added a huge number of asserts
extended database test methods


git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3579 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2007-04-19 13:37:02 +00:00
orbiter
ca79362b9d disabling auto-setting of remote crawl performance 
see also http://www.yacy-forum.de/viewtopic.php?t=3849

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3577 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2007-04-18 18:36:06 +00:00
orbiter
7a7a1c7c29 fight against problems with remove-methods and synchronization 
- some bugs may have been fixed with wrong removal operations
- removed temporary storage of remove-positions and replaced by direct deletions
- changed synchronization
- added many assets
- modified dbtest to also test remove during threaded stresstest

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3576 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2007-04-17 15:15:47 +00:00
orbiter
b6a5f53020 removed double synchronization from kelondroRecords.USAGE to prevent thread locking. 
The method synchronization should be sufficient

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3574 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2007-04-15 21:13:54 +00:00
(no author)
6186185775 *) Moved some comments to javadoc 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3573 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2007-04-14 10:11:37 +00:00
orbiter
063063aa0c fix for 100% cpu bug during dht selection 
see also: http://www.yacy-forum.de/viewtopic.php?p=34068#34068

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3570 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2007-04-13 13:40:19 +00:00
michitux
4990909178 Some bugfixes, new layout/style for image search results: 
* removed divide by zero bug when 20_dhtdistribution_busysleep is 0
 * replaced German comment with wrong charset in source/de/anomic/plasma/plasmaCrawlBalancer.java by an English one
 * replaced the table-fix for floating behind snipped images by a br with clear
 * removed unnecessary old xhtml-files (were not in use, they were created when we weren't having xhtml for testing)
 * new layout for image-search results: replaced the old one with spans and tables inside (not valid) with new divs, now each image snippet container has the same size
TODO:
 * the ids of the snippetLoading-divs aren't valid because ids must start with an alphabetic letter or an underscore, they have to be prefixed
 * in the returned snippet-xml is an unresolved pattern for status (the status is only set for text snippets)

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3566 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2007-04-12 18:21:17 +00:00
orbiter
78d04bcbcf fixed bug in search statistics 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3562 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2007-04-12 09:24:56 +00:00
orbiter
b79b4082e2 completed search exclusion: 
- exclusion on index-level (not only from search snippets)
- exclusion hand-over at remote search protocol

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3556 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2007-04-10 12:27:03 +00:00
auron_x
d66b0276e3 *) removed log-output for PPM-calc 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3553 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2007-04-07 10:03:12 +00:00
low012
4400acc27d *) created new 8 bit oldschool style font for possible future use 
*) main method is generalization of main method of ymageFontGenerator:
   it does not matter how many lines of how many bits a font is made of
   as long as the values stay the same within the font -> use this class as 
   a template for your own font generators and be a happy camper
*) main method checks if font is valid (96 characters, all letters must have
   same number of lines and same number of bits per line)
*) ***** I have not checked if the result is really a valid font so far. *****


git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3552 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2007-04-05 21:05:16 +00:00
orbiter
25070822a5 fix for http://www.yacy-forum.de/viewtopic.php?p=33925#33925 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3551 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2007-04-05 19:08:59 +00:00
orbiter
06a7978730 moved url pattern matching for search to better place 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3550 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2007-04-05 15:08:14 +00:00
orbiter
159bd0cab5 diverses; b.o. fix for http://www.yacy-forum.de/viewtopic.php?p=33914#33914 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3549 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2007-04-05 14:58:29 +00:00
orbiter
cdc7b77a62 fix for http://www.yacy-forum.de/viewtopic.php?p=33916#33916 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3548 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2007-04-05 14:47:45 +00:00
orbiter
40c14a4f0e - better implementation of search query properties 
- basic protection against start-up problems when database files are corrupted
- auto-delete of not-critical databases during startup when load error occurs
- on-the-fly reset option for all database tables
- automatic on-the-fly reset for seed tables during enumeration exceptions

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3547 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2007-04-05 10:14:48 +00:00
theli
1696606b7f *) changing loglevel of "PPM-Calculation" message 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3545 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2007-04-04 13:09:59 +00:00
auron_x
f30bf1683e *) corrected spelling of captcha 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3544 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2007-04-04 10:13:58 +00:00
orbiter
fcdf000fbc bugfix for http://www.yacy-forum.de/viewtopic.php?p=33838#33838 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3543 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2007-04-03 22:08:40 +00:00
low012
d7edc9740b *) added correct (c) and Last*-data 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3542 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2007-04-03 20:58:45 +00:00
low012
ee241f32e6 *) very basic capcha class (see coding sections of forum for more details) 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3541 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2007-04-03 20:54:14 +00:00
orbiter
6e7340ef52 added exclusion search 
(you can now search and exclude words from the result with '-')

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3540 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2007-04-03 15:35:29 +00:00
orbiter
e4734a8b6b fix for fix in SVN 3537 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3539 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2007-04-03 14:42:29 +00:00
orbiter
356033aceb fixed bug with continuous reset of balancer file index 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3537 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2007-04-03 12:36:24 +00:00