Marown/yacy_search_server
1
0
Fork 0
mirror of https://github.com/yacy/yacy_search_server.git synced 2024-09-21 00:00:13 +02:00
Code Issues Actions Packages Projects Releases Wiki Activity
3,055 Commits 1 Branch 20 Tags 836 MiB
e1fb3550ab
Commit Graph

2006 Commits

Author SHA1 Message Date
orbiter
7cc4cec9c9 bugfix for assertion bugs documented in 
http://www.yacy-forum.de/viewtopic.php?p=28261#28261

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3030 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2006-12-01 01:30:05 +00:00
orbiter
97596af843 patch for wrong old RWI entries 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3028 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2006-11-30 14:24:41 +00:00
hydrox
2c69cc969a *) more special chars removed 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3027 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2006-11-30 07:39:09 +00:00
hydrox
ebb42906f8 *) removed special characters 
*) added Copyright comments

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3026 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2006-11-30 07:33:54 +00:00
theli
954db729db *) Bugfix for ArrayIndexOutOfBoundsException during SSL detection 
See: http://www.yacy-forum.de/viewtopic.php?p=28247#28247

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3025 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2006-11-30 06:38:53 +00:00
orbiter
5afd2f283c fix for http://www.yacy-forum.de/viewtopic.php?p=27633#27633 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3024 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2006-11-30 01:17:28 +00:00
orbiter
9dfae90da8 better control of sortBound; should help to avoid an indexOutOfBound Exception 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3023 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2006-11-30 00:24:12 +00:00
orbiter
4ce590622f - more asserts 
- better memory usage during remove in kelondroRowSet

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3022 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2006-11-30 00:23:07 +00:00
orbiter
7dbcd358b4 fix for http://www.yacy-forum.de/viewtopic.php?p=28231#28231 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3021 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2006-11-29 23:09:56 +00:00
orbiter
86394e7a56 fix for cache-delete problem: 
- better synchronization
- files are only deleted if they have been in the cache for 5 minutes
- hash-path for the HTCACHE is now default

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3018 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2006-11-29 00:34:25 +00:00
orbiter
ceb9e3aa17 - enhanced parser: collection of audio, video, image and application links 
- enhanced condenser: better handling of utf-8 and pre-formatted texts


git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3017 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2006-11-28 15:00:15 +00:00
orbiter
984285bdd6 better organisation of dns hit/miss cache flush 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3016 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2006-11-27 15:53:42 +00:00
orbiter
73c63578ad - activated the dns miss cache 
- added a cache-control for cache miss flush to the dns miss cache
- better naming of cache variables to distinguish hit- and miss- cache

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3015 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2006-11-27 15:27:43 +00:00
theli
eb20ec3837 *) soap-service: adding function to check if a specific url is blacklisted 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3014 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2006-11-26 08:53:43 +00:00
orbiter
0b9370a9dc fix for http://www.yacy-forum.de/viewtopic.php?p=28108#28108 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3013 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2006-11-25 23:33:28 +00:00
orbiter
b5a29e9651 - fix for snippets that are too short 
- added keyword to snippet fetch to suppres removal of not-found snippet words (for debugging)


git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3009 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2006-11-25 00:38:09 +00:00
orbiter
13868ebb3e reverted IP-patch in yacySeed 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3007 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2006-11-24 16:23:08 +00:00
allo
4cb688018d wikiAdmin Recht 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3006 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2006-11-24 11:32:00 +00:00
low012
29fa17bd40 *) simplified some code in wikiCode.java 
*) deleted outdated text in Settings_p-html (see http://www.yacy-forum.de/viewtopic.php?p=28027)


git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3005 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2006-11-24 02:52:38 +00:00
orbiter
f1528672b1 filtering of non-index pages during index-of search 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3004 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2006-11-24 02:46:02 +00:00
orbiter
0a0c3edeb6 fixed a bug in index transfer 
- the encoding within the new entry format for binary data was wrong
- the string parser of RWI receive had to be enhanced

added some mor debugging tools
- a target peer for index transfer can now be selected by typing in the peer name
- the RWI result list has an entry counter

enhanced routing
- if communication is between two peers that have the same IP address,
  the loopback address 127.0.0.1 is used instead the public IP
  to contact the peer

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3003 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2006-11-24 01:12:14 +00:00
orbiter
8e7215475b - extended ViewFile to use is as debugging-tool: you can now use the 
post-parameter url to submit an url directly
- fixed some bugs in text parser (not all parts had been analysed)
- fixed a bug in remote search interface (could not handle constraints)

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3001 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2006-11-23 15:47:19 +00:00
orbiter
30888e7a2f implementation of search constraints 
Such constraints may formulate specific restrictions to web searches
This is implemented by scraping information for constraints from a web
page during parsing, and storing flags to the pages within the web index.

In this first step, only information for index pages ("index of", directory listings)
are scraped and stored in flags
- added new flag class kelondroBitfield
- added scraper method in condenser
- added bitfield structure for all scrape types (see also condenser)
- added bitfield structure for appearance locations (see RWIEntry)
- added handover protocol for remote search and index distribution
- extended kelondroColumn class to hold bitfield types
- added another search attribute on search page (index.html)
- extended search-filter to enable filtering of non-matching constraints
- set all new database types to be default
- refactoring: moved word hash generation to condenser class

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2999 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2006-11-23 02:16:30 +00:00
orbiter
49a83f99d9 - fix for wrong DHT ordering in DHT selection 
- fix for http://www.yacy-forum.de/viewtopic.php?t=3112&highlight=

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2995 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2006-11-21 00:36:41 +00:00
orbiter
e55ef0df28 - automatic migration of old RWI entries to new format during remote search 
if new collections are activated
- one more assert in RowSet, control of removeMarker

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2993 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2006-11-20 22:55:27 +00:00
orbiter
f4b547dc13 limited index transfer to peer with version 0.486 
this protects peers with version below 0.486 from new RWI objects
(which they cannot handle)

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2988 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2006-11-20 02:46:53 +00:00
orbiter
10a4ab5195 disabled some (more) write caches 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2987 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2006-11-20 00:27:02 +00:00
orbiter
09bcc10344 bugfix for some problems of last change with assortments 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2986 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2006-11-19 23:10:58 +00:00
orbiter
e3d75f42bd final version of collection entry type definition 
- the test phase of the new collection data structure is finished
- test data that had been generated is void. There will be no migration
- the new collection files are located in DATA/INDEX/PUBLIC/TEXT/RICOLLECTION
- the index dump is void. There will be no migration
- the new index dump is in DATA/INDEX/PUBLIC/TEXT/RICACHE

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2983 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2006-11-19 20:05:25 +00:00
theli
58d79a9ac1 *) more error logging 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2974 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2006-11-19 13:28:09 +00:00
theli
73552db482 *) bugfix adding missing lib names to copy and zip commands 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2971 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2006-11-18 16:52:17 +00:00
theli
5c0669429e *) soap: adding function to query the peer list 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2968 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2006-11-17 19:26:55 +00:00
orbiter
c9364246cc introduced new RWI-Object. 
This will be used for the final version of the collections.
The new object is not yet used.

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2966 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2006-11-17 14:17:20 +00:00
hydrox
f442af956c *) first version of build-in logalizer 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2965 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2006-11-17 11:49:21 +00:00
theli
9ea9a9f157 *) bugfix for bookmark service: get didn't work when using date=null 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2963 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2006-11-16 16:20:00 +00:00
orbiter
b94ee53958 some enhancements to url retrieval 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2959 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2006-11-14 01:19:24 +00:00
theli
203f2bde9a *) adding function to query the pause/resume state of the crawling queues 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2958 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2006-11-13 06:25:15 +00:00
orbiter
d34f10c63d some tests with reverse dns lookup 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2954 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2006-11-12 00:28:10 +00:00
(no author)
24ac4e8860 Bugfix to "-UNRESOLVED_PATTERN- bei Hostname-Änderung" (http://www.yacy-forum.de/viewtopic.php?t=3093) 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2952 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2006-11-11 17:52:20 +00:00
orbiter
e628d34e16 patches for bad data 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2951 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2006-11-11 14:35:36 +00:00
orbiter
497428c8ec refactoring 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2949 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2006-11-10 01:13:33 +00:00
(no author)
a51417d86b Bugfix: language of ConfigLanguage_p.html was not changed properly when a different language was choosen here 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2948 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2006-11-09 22:18:16 +00:00
theli
4147c9f490 *) better error logging on soap engine initialization 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2946 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2006-11-09 17:20:34 +00:00
orbiter
76fceb9997 refactoring 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2945 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2006-11-09 16:32:34 +00:00
theli
f77d624b94 *) bugfix for persistent connection support on transfer-encoded requests 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2942 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2006-11-09 05:59:56 +00:00
orbiter
eeda881553 bugfix for last commit 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2938 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2006-11-08 16:38:19 +00:00
orbiter
bb7d4b5d5e refactoring to prepare new RWI entry object 
- moved all url and index(RWI) entries to index package
- better naming to distinguish RWI entries and URL entries


git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2937 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2006-11-08 16:17:47 +00:00
orbiter
6412c926bc new url entry interface 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2936 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2006-11-08 15:29:20 +00:00
orbiter
bdc9216366 - more asserts 
- some bugfixes
- some patches for bugs that are already in the database

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2935 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2006-11-08 02:08:33 +00:00
low012
b18273bbb2 *) YaCy will extract URLs from <embed> tags now. 
*) YaCy will extract URLs from <param> tags now if the name of the parameter is "movie". 


git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2934 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2006-11-07 20:42:37 +00:00
orbiter
1751a799ac - deactivated all write buffers 
- fixed a storage bug


git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2933 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2006-11-07 10:56:36 +00:00
orbiter
ba967c4875 - bugfixes and debug code 
- ne generalized index class indexCachedRI

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2930 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2006-11-07 01:09:02 +00:00
orbiter
eaad91d84f fixed wrong RAM calculation 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2928 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2006-11-06 15:53:42 +00:00
orbiter
ee4715a21c - more asserts 
- bugfix for performaceMemory
- refactoring of index ram cache: renamed indexRAMCacheRI to indexRAMRI, to make space for a cached indexRI, which should be named indexRAMCacheRI

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2925 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2006-11-06 10:15:05 +00:00
orbiter
114a76a86e - added flag to urlhash that shows that domain is a local domain 
- enhanced local domain detection
- bugfixing for memory assignment in kelondroFlexSplit
- automatic memory assignment to caches according to available RAM
- bugfixes for details during search process

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2924 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2006-11-06 02:05:39 +00:00
orbiter
eafb5ecd22 - better usage of memory resources for kelondroFlexSplit 
- kelondroFlexTables does always load a RAM cache if it has enough
  ram assigned. Othervise it creates a kelondroTree file-index.
  If more memory is re-assigned, the file-index is deleted again,
  and RAM is used. Beware that assignement of too less RAM forces
  creation of file indexes and start-up time may last for hours.

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2923 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2006-11-05 21:30:53 +00:00
orbiter
b2d51be33c bugfix for latest changes to entry generalization 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2922 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2006-11-05 19:07:19 +00:00
(no author)
e59ff8b657 Bugfix: language of ConfigBasic.html was not changed properly when a different language was choosen here. Note: there's a similair bug on ConfigLanguage_p.html 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2921 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2006-11-05 17:43:37 +00:00
theli
892b9f2fc4 *) additional soap function to query peer status 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2920 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2006-11-05 16:46:32 +00:00
theli
bd3710a974 *) new xml template to view peer profile as xml 
*) bugfix for wrong profile display (some fields where displayed twice)
*) new soap functions to get and set peer profile

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2919 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2006-11-05 16:26:25 +00:00
theli
d1afe1ce6b *) adding xml template to get the message list as xml 
*) Bugfix in client stub jar generation (too many files where added)
*) new soap service to manage peer messages

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2918 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2006-11-05 15:18:33 +00:00
orbiter
d454ca44ee update of cache logging 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2917 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2006-11-05 14:48:21 +00:00
hermens
8385557672 Small fix for the Cache Monitor when using proxyCacheLayout=hash 
see: http://www.yacy-forum.de/viewtopic.php?p=27394#27394



git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2916 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2006-11-05 14:35:35 +00:00
theli
f37e2041e8 *) adding soap function to import yacy bookmarks from xml or html (transfered via soap attachments) 
*) soapHandler: code cleanup for service deployment

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2915 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2006-11-05 09:56:39 +00:00
orbiter
f1ed55a5fc bugfix for last commit 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2913 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2006-11-05 02:23:32 +00:00
orbiter
8fdefd5c68 generalization of payload definition of index storage 
this is one step forward to the migration to a new collection data format

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2912 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2006-11-05 02:10:40 +00:00
theli
29a1f132ec *) some strings replaced by constants 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2910 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2006-11-04 16:33:02 +00:00
theli
4a3ec63e34 *) new soap service to manage yacy bookmarks 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2906 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2006-11-04 13:47:43 +00:00
(no author)
9b3fd2b9e5 *) removing doctype definition to avoid problems with xml parser 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2905 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2006-11-03 19:48:33 +00:00
(no author)
c64d5018b4 *) Bugfix. Problem in XML Parser 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2903 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2006-11-03 19:34:31 +00:00
theli
5e57e0814d *) new soap function to display log 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2902 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2006-11-03 14:39:48 +00:00
theli
ad248d61ca *) more verbose exception 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2901 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2006-11-03 14:37:12 +00:00
hydrox
7e8669b15c *) added possibility to "recycle" a DHTChunk that failed to transfer. 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2898 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2006-11-02 21:32:59 +00:00
low012
4feaa91890 *) Added additional MIME-Type. 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2895 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2006-11-02 13:32:04 +00:00
low012
89af433879 *) Deleted parts of WebCat that were not needed for parsing SWFs. 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2893 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2006-11-02 11:13:33 +00:00
orbiter
46a712e195 - more asserts 
- simplified indexURLEntry

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2891 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2006-11-01 14:00:15 +00:00
low012
8c9bc7e341 *) extracting urls works now 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2890 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2006-11-01 09:22:15 +00:00
orbiter
fc2936d500 bugfix for internal index entry generation 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2889 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2006-11-01 08:55:49 +00:00
low012
493391e42d *) new flash parser, still experimental 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2888 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2006-11-01 00:52:42 +00:00
orbiter
215c4e65f1 code cleanup 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2887 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2006-10-31 22:10:25 +00:00
orbiter
bd4f43cd66 - fixed a null pointer exception bug 
- switched off more write caches
- re-enabled index-abstracts search

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2885 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2006-10-31 02:45:41 +00:00
auron_x
194d42b6a7 *) changed PPM-calculation to be more accurate 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2884 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2006-10-30 19:04:06 +00:00
orbiter
fe8afaf426 switched off usage of write cache for imprortant databases 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2883 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2006-10-30 02:59:22 +00:00
orbiter
985fd807cc bugfixing in collection methods 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2882 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2006-10-30 02:39:39 +00:00
theli
c7bea4addb *) soap api 
- adding function to get and set message forwarding
   - adding new testclass 


git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2878 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2006-10-29 08:40:48 +00:00
theli
ee4d4e8567 *) Soap-handler: bugfix. wrong content-length was send when using content-encoding 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2877 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2006-10-29 08:39:48 +00:00
orbiter
d3431433b0 more anonymization in logging 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2876 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2006-10-28 22:28:33 +00:00
orbiter
e6044e5198 bugfix for 
http://www.yacy-forum.de/viewtopic.php?p=27207#27207
and
http://www.yacy-forum.de/viewtopic.php?p=27219#27219

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2875 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2006-10-28 21:43:12 +00:00
theli
4d19d94348 *) bugfix for nullpointerexception 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2874 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2006-10-28 13:49:39 +00:00
theli
532c23b5c7 *) soap handler 
- better errorhandling 
   - adding support for outgoing transfer- and content-encoding
   - avoid holding outgoing messages into memory before sending them

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2872 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2006-10-28 12:31:48 +00:00
orbiter
78b7f6f7fd bugfix for index remove bug, 
appeared after search where snippet-loading triggered word removal

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2869 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2006-10-28 00:22:10 +00:00
(no author)
0e79f2fd7e name of the file to tranlate apears ahead its translation 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2868 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2006-10-27 23:51:57 +00:00
orbiter
ebd2d629d8 added missing file for last commit 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2866 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2006-10-26 13:53:00 +00:00
orbiter
147d88cf23 re-design of database caching 
this should reduce IO a lot, because write caches are now actived for all databases
- added new caching class that combines a read- and write-cache.
- removed old read and write cache classes
- removed superfluous RAM index (can be replaced by kelonodroRowSet)
- addoped all current classes that used the old caching methods
- more asserts, more bugfixes


git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2865 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2006-10-26 13:50:50 +00:00
orbiter
4e363108e1 - removed bad debug code that caused a large and unnecessary delay during global search 
- fixed problem that global search results disappear after a search
- removed some stopwords

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2861 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2006-10-25 02:24:41 +00:00
orbiter
f21ede312e bugfixes for internals of database organization 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2860 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2006-10-25 01:21:05 +00:00
orbiter
eb4bfb0e9d fixed problem with cache.profile() 
see also: http://www.yacy-forum.de/viewtopic.php?p=27109#27109

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2859 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2006-10-24 22:34:13 +00:00
orbiter
2a9d868f6d - removed object cache from kelondroTree 
- generalized object caching and added new object caching class
- added object caching wherever kelondroTree was used
- added object caching also to usage of kelondroFlex
- added object buffering (a write cache) to NURLs
- added many assert statements; fixed bugs here and there
- added missing close methods to latest added classes

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2858 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2006-10-24 13:48:16 +00:00
theli
7299dc30e3 *) new soap service to manage the yacy file-share 
- upload / download files (as soap attachment)
   - create directory
   - receive directory listing
   - delete files / directories
   - change file comment

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2857 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2006-10-24 12:15:56 +00:00
theli
777e39cea0 *) new template to display the dir-listing in xml format. 
This can e.g. be done by using the url http://localhost:8080/share/?format=xml

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2856 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2006-10-24 12:13:46 +00:00
theli
9e8942a064 *) adding method to implement blacklist from file 
- file transfer is done via soap attachments (see BlaclistSerivceTest for details)

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2855 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2006-10-24 06:18:19 +00:00
theli
4d1f933ea1 *) avoid reading of content body into memory 
*) Bugfix for soap attachment support

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2854 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2006-10-24 06:15:03 +00:00
theli
88cfdecd38 *) Bugfix: calling close must not close the wrapped input stream, otherwise 
keep-alive connections would terminate

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2853 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2006-10-24 06:09:38 +00:00
theli
d38ef0493d *) be more tolerant against missing ports in url 
"http://yacy.net:/" is now interpreted as "http://yacy.net/"
   See: http://www.yacy-forum.de/viewtopic.php?p=27102

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2852 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2006-10-24 05:22:54 +00:00
theli
cfe54fedc7 *) Bugfix for resolveBackpath problem with tailing /.. 
*) Junit testclass for resolveBackpath testing 

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2850 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2006-10-24 05:07:34 +00:00
orbiter
dc056fabf3 small bugfix 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2847 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2006-10-23 01:22:50 +00:00
orbiter
278d8c3c7e - more asserts 
- bugfix for reading of previously deleted nodex

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2845 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2006-10-23 00:59:55 +00:00
allo
5a6488256d catch the "username too short" exception 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2844 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2006-10-22 21:40:16 +00:00
orbiter
2d3f1a53fd handling of Missing byte-order mark exception 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2842 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2006-10-22 12:18:15 +00:00
theli
ac13fa763a *) bugfix for blacklist remove (blacklist was not informed about remove) 
*) adding new soap service class for blacklist management
*) new junit class to test soap blacklist service

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2841 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2006-10-22 08:32:55 +00:00
allo
8a5c2d0a19 fix for supertemplates, too. 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2839 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2006-10-21 16:53:31 +00:00
allo
c35793fb46 fix for last commit 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2838 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2006-10-21 16:41:22 +00:00
theli
3e0516446b *) new soap function to get the current queue status 
*) new junit testclass to test soap statusService
*) refactoring of admin service (usage of constants instead of strings)
*) libraries upgraded to newer version + adding missing dependency

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2836 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2006-10-21 15:11:01 +00:00
allo
a831c83025 create servletProperties, with the servlet specific funktions from serverObjects 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2835 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2006-10-21 15:01:53 +00:00
orbiter
83a0efc65a better assert statements and fixes 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2833 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2006-10-21 10:50:30 +00:00
karlchenofhell
d13b381f83 - added mint-green skin 
- removed test-urls because of problems with text-encoding

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2832 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2006-10-21 10:24:32 +00:00
orbiter
2025e885d6 a fix for problems with remove situations in kelondroFlexSplitTable 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2831 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2006-10-21 00:19:00 +00:00
theli
b12da510f3 *) adding optional libraries for needed for soap attachments 
(jikes won't compile without them)

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2827 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2006-10-20 14:09:20 +00:00
theli
9eecc9a888 *) libs added to classpath 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2824 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2006-10-20 13:00:52 +00:00
theli
a1acc9c389 *) new function to configure distributed crawling 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2823 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2006-10-20 12:48:19 +00:00
theli
0996e550e7 *) deploy soap peer admin service 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2822 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2006-10-20 04:45:56 +00:00
orbiter
3ffc5b8793 fixed problem with serverCharBuffer.append(char) 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2821 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2006-10-19 21:44:02 +00:00
orbiter
8b56887676 removed unused code 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2820 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2006-10-19 21:30:02 +00:00
orbiter
06854988da - full integration of new LURL database in INDEX 
- added migration method for urlHash.db into INDEX

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2819 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2006-10-19 21:14:37 +00:00
(no author)
02c66c04f2 *) Missing file from last commit 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2818 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2006-10-19 17:58:36 +00:00
octoate
e4a3574b77 StringBuffer now resets every time the parser is called 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2817 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2006-10-19 16:58:45 +00:00
theli
ef912811f1 *) adding new soap service for peer administration 
- configure dht transfer properties
   - configure remote proxy
   - configure peer name / peer port
   - configure admin username + pwd
   - get peer version information
   - set/get peer configuration settings
   - shutdown peer
*) new function to get the opensearch description via soap call

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2816 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2006-10-19 16:07:59 +00:00
karlchenofhell
ce237aefad - assortment-sizes table from PerformanceQueues_p.html is not shown if not used 
- escape query- and fragment-part of an url as well
- new resolveBackpath for urls: http://www.yacy-forum.de/viewtopic.php?t=2679#24867

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2815 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2006-10-19 15:27:24 +00:00
theli
68204ff729 *) Suppressing for bad client requests. 
See: http://www.yacy-forum.de/viewtopic.php?p=26918

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2814 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2006-10-19 11:10:56 +00:00
theli
c1dff41f99 *) adding possibility to deploy custom SOAP services 
See: http://www.yacy-forum.de/viewtopic.php?p=26748#26748

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2813 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2006-10-19 10:56:53 +00:00
theli
df49724f28 *) better error handling for seed upload - test download - problems 
See: http://www.yacy-forum.de/viewtopic.php?p=26814#26814

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2812 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2006-10-19 10:10:53 +00:00
theli
a5b9b514c1 *) retry crawling without content-encoding if the content-encoding header was not correct 
See: http://www.yacy-forum.de/viewtopic.php?p=26917#26917

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2811 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2006-10-19 08:45:52 +00:00
theli
52466067d8 *) Bugfix for ArrayIndexOutOfBoundsExceptions which occure because SimpleDateFormat is not thread-safe 
See: http://www.yacy-forum.de/viewtopic.php?t=2995

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2810 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2006-10-19 08:33:53 +00:00
theli
b357a13e9a *) adding synchronization block because SimpleDateFormat is not thread-safe 
See: http://www.yacy-forum.de/viewtopic.php?p=26906#26906

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2809 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2006-10-19 07:48:13 +00:00
theli
92f774edd1 *) Better charset encoding detection 
*) New testclass for charset encoding detection tests

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2808 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2006-10-19 07:02:18 +00:00
orbiter
b79e06615d - added new LURL.Entry class for next database migration 
- refactoring of affected classes

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2802 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2006-10-18 22:25:07 +00:00
octoate
cc24dde5e0 First version of a MS Excel parser based on Apache POI 
(event based parsing)

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2801 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2006-10-18 19:13:37 +00:00
karlchenofhell
4c63129136 - stupid mistake... 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2798 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2006-10-18 15:14:38 +00:00
karlchenofhell
b14a500b88 - removed debug output from PerformanceMemory_p 
- added URL escaping (tested, nevertheless watch out for possibly broken URLs)

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2797 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2006-10-18 14:51:37 +00:00
karlchenofhell
ebf0da2a45 - now the fix http://www.yacy-forum.de/viewtopic.php?t=2974 works 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2796 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2006-10-18 12:07:17 +00:00
theli
09337c9751 *) Bugfix wrong chars in soap search result document 
See: http://www.yacy-forum.de/viewtopic.php?t=2906

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2795 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2006-10-18 05:15:50 +00:00
theli
3d152bfe43 *) Logging message added 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2794 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2006-10-18 04:23:00 +00:00
karlchenofhell
b5e40e2fa2 - fix for http://www.yacy-forum.de/viewtopic.php?t=2974 (no cache-sizes for new db) 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2792 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2006-10-17 21:01:35 +00:00
theli
96f45e9b15 *) Bugfix wrong chars in soap search result document 
See: http://www.yacy-forum.de/viewtopic.php?t=2906

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2791 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2006-10-17 13:45:00 +00:00
theli
da2ac6fa23 *) adding new ant target to allow generation of client stub classes for yacy soap api 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2789 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2006-10-17 09:48:09 +00:00
theli
a9cc6df21b *) adding wsdl files to generate client stub classes with ant 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2788 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2006-10-17 09:47:04 +00:00
orbiter
77a59a115d refactoring of indexing methods 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2787 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2006-10-16 15:04:16 +00:00
orbiter
14490f0a83 added missing flush statement 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2786 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2006-10-16 09:42:35 +00:00
orbiter
688cbfb776 - bugfixing for flextable bug 
- bugfixing for collection index bug
- several other bugfixes

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2785 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2006-10-16 00:27:25 +00:00
allo
a29b4d4fb5 extended Supertemplates for Headerincludes. 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2780 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2006-10-15 13:43:46 +00:00