Marown/yacy_search_server
1
0
Fork 0
mirror of https://github.com/yacy/yacy_search_server.git synced 2024-09-21 00:00:13 +02:00
Code Issues Actions Packages Projects Releases Wiki Activity
2,598 Commits 1 Branch 20 Tags 836 MiB
440c6ee657
Commit Graph

118 Commits

Author SHA1 Message Date
hermens
440c6ee657 Implement alternative htcache layout 
mostly according to: http://www.yacy-forum.de/viewtopic.php?p=26205#26205



git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2718 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2006-10-08 17:25:19 +00:00
theli
f17ce28b6d *) plasmaHTCache: 
- method loadResourceContent defined as deprecated. 
     Please do not use this function to avoid OutOfMemory Exceptions 
     when loading large files
   - new function getResourceContentStream to get an inputstream of a cache file
   - new function getResourceContentLength to get the size of a cached file
*) httpc.java:
   - Bugfix: resource content was loaded into memory even if this was not requested
*) Crawler:
   - new option to hold loaded resource content in memory
   - adding option to use the worker class without the worker pool 
     (needed by the snippet fetcher)
*) plasmaSnippetCache
   - snippet loader does not use a crawl-worker from pool but uses
     a newly created instance to avoid blocking by normal crawling
     activity.
   - now operates on streams instead of byte arrays to avoid OutOfMemory 
     Exceptions when operating on large files 
   - snippet loader now forces the crawl-worker to keep the loaded
     resource in memory to avoid IO 
*) plasmaCondenser: adding new function getWords that can directly operate on input streams
*) Parsers
   - keep resource in memory whenever possible (to avoid IO)
   - when parsing from stream the content length must be passed to the parser function now.
     this length value is needed by the parsers to decide if the parsed resource content is to large
     to hold it in memory and must be stored to file 
   - AbstractParser.java: new function to pass the contentLength of a resource to the parsers
   


git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2701 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2006-10-03 11:05:48 +00:00
orbiter
df1629b05a - code cleanup 
- version 0.471
- moved surftipps to own web page


git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2676 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2006-09-29 22:27:20 +00:00
theli
625c2ce6b1 *) bugfix for snippet fetching problem if content but not http header is available in cache 
See: http://www.yacy-forum.de/viewtopic.php?p=25748

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2651 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2006-09-22 11:55:28 +00:00
theli
1dc12d6659 *) Bugfix for shutdown problem caused by cacheScan thread 
See: http://www.yacy-forum.de/viewtopic.php?p=25729

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2636 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2006-09-20 04:36:25 +00:00
orbiter
b251076e64 avoid ConcurrentModificationException 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2563 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2006-09-13 10:36:18 +00:00
theli
dae763d8e3 git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2495 6c8d7289-2bf4-0310-a012-ef5d649a1542 2006-09-06 14:31:17 +00:00
theli
ffbf416e76 *) direct access to requestheader of htCache.Entry removed to make it more http independent 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2486 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2006-09-04 15:29:45 +00:00
theli
393a7d10be *) setting htCache.Entry fields to private 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2484 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2006-09-04 15:03:54 +00:00
theli
ab5a9bee66 *) adding some copyright headers 
*) next step of restructuring for new crawlers
   - adding first testversion of ftp crawler class
   -- does not create a htCache entry yet

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2483 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2006-09-04 14:38:29 +00:00
orbiter
db1eae0227 * simplified initialization of database objects 
* replaced kelondroTree for NURLs by kelondroFlex
* replaced kelondroTree for EURLs by kelondroFlex
take care, may be very buggy
please finish crawls before updating. crawls will be lost.

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2452 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2006-08-24 02:19:25 +00:00
hermens
0b73f2b132 Repair DNS prefetch during cacheScan 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2451 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2006-08-24 01:31:08 +00:00
orbiter
23dd972608 fixed memory calculation in performanceMemory web page 
fixed also maximum cache size computation

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2429 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2006-08-20 01:20:34 +00:00
orbiter
abf22f6e60 removed url normalform computation from htmlFilterContentScraper. 
This method was implemented in de.anomic.net.URL


git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2377 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2006-08-11 15:09:22 +00:00
orbiter
3879a0ecd0 replaced java.net.URL usage by use of new class de.anomic.net.URL 
This shall be seen as an experiment to exclude all cases where
there could be a DNS lookup during URL comparisment.

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2290 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2006-07-13 01:21:53 +00:00
orbiter
92f4cb4d73 added option to configure the start-up delay time for kelondro database files. 
the start-up delay is used to pre-load the database node cache

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2276 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2006-07-03 23:57:33 +00:00
orbiter
66964dc015 removed high/med/low from kelondroRecords cache control. 
this was done because testing showed that cache-delete operations
slowed down record access most, even more that actual IO operations.
Cache-delete operations appeared when entries were shifted from low-priority
positions to high-priority positions. During a fill of x entries to a database,
x/2 delete situation happen which caused two or more delete operations.
removing the cache control means that these delete operations are not
necessary any more, but it is more difficult to decide which cache elements
shall be removed in case that the cache is full. There is not yet a stable
solution for this case, but the advantage of a faster cache is more important
that the flush problem.

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2244 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2006-06-25 10:31:38 +00:00
orbiter
4d8f8ba384 added cache-performance analysis for node caches 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2140 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2006-05-24 09:40:08 +00:00
orbiter
90d569d70f refactoring of index management: 
url storage is part of index management; moved plasmaURL to indexURL

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2122 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2006-05-19 23:50:55 +00:00
orbiter
a474669338 start with refactoring of index management 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2110 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2006-05-16 16:11:55 +00:00
auron_x
55ea4cbfe6 *)reverted patch for memory-display issue 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2095 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2006-05-14 18:09:28 +00:00
auron_x
53d9ab6db7 *)fixed bug in PerformanceMemory_p.java which caused negative memory-values on big peers 
see http://www.yacy-forum.de/viewtopic.php?t=2370

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2091 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2006-05-14 08:42:41 +00:00
orbiter
29b1b0823c added monitoring of new object cache to performanceMemory page 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2072 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2006-05-09 10:03:12 +00:00
borg-0300
30e4fc39a5 HTCache extended 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2015 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2006-04-08 13:05:04 +00:00
borg-0300
da6a8bafa2 rename currCacheSize -> curCacheSize; 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2010 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2006-04-07 13:15:15 +00:00
orbiter
128e4ab199 - in serverSystem: maxPathLength is now a variable, not a method 
- upon startup the calculated maximum path length is shown

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1932 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2006-03-19 01:33:20 +00:00
orbiter
30e3e3a0fd adopted MAXPATHLENGTH to host system capabilities 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1930 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2006-03-19 00:29:27 +00:00
borg-0300
8865948e4e Cleanup; 
Methode replaceRegex added;
Constant MAXPATHLENGTH added;

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1923 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2006-03-18 13:34:32 +00:00
borg-0300
5f6fdf1786 Bugfix for getCachePath(URL url) 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1909 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2006-03-17 16:39:29 +00:00
theli
759800f543 *) Bugfix for storeHTCache problem 
- content was not indexed if storeHTCache was off
   See: http://www.yacy-forum.de/viewtopic.php?p=18269
   See: http://www.yacy-forum.de/viewtopic.php?t=1882
   See: http://www.yacy-forum.de/viewtopic.php?t=241

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1800 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2006-03-03 08:30:08 +00:00
borg-0300
139ba4e0c8 Bugfix for getCachePath(URL url) 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1510 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2006-02-01 19:07:49 +00:00
orbiter
3419b3bcdd fix for bug that caused the peer-counter problem. 
See http://www.yacy-forum.de/viewtopic.php?p=16016#16016
The kelondroDyn now uses a generic fill character.
kelondroDyn-Tables containing peer/word/url-hashes must not use '_'
as fill character.

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1498 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2006-01-30 22:16:55 +00:00
rramthun
6c02f889f7 Cosmetic changes. 
Corrected version numbering as described in http://www.yacy-websuche.de/wiki/index.php/De:Versionsnummern

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1453 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2006-01-26 15:12:44 +00:00
hermens
861aae678d *) cleanup cacheAge database when cleaning up the HTCache 
*) Log directory deletes with level Fine



git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1427 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2006-01-24 14:21:01 +00:00
theli
ecdc1f7547 *) Bugfix for crawling URLs with query parameters 
See: http://www.yacy-forum.de/viewtopic.php?p=14065
*) Preparation for http://www.yacy-forum.de/viewtopic.php?t=1719

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1405 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2006-01-22 16:39:10 +00:00
theli
7c22afe3de *) Bugfix for NullpointerException in deleteOldHTCache 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1326 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2006-01-13 10:37:42 +00:00
borg-0300
9d8dca750e BUGFIX for my last commit 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1306 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2006-01-10 10:38:11 +00:00
borg-0300
5449193167 bugfix for http://www.yacy-forum.de/viewtopic.php?t=1706 (i hope) 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1304 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2006-01-09 23:19:05 +00:00
borg-0300
2a23f5d419 F..., Sorry, no time, later 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1303 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2006-01-09 19:26:39 +00:00
borg-0300
3a2d13786e bugfix for http://www.yacy-forum.de/viewtopic.php?t=1706 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1302 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2006-01-09 19:18:41 +00:00
borg-0300
b24fcc8ca4 oom 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1281 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2006-01-03 11:08:45 +00:00
borg-0300
7da232b5b9 HTCache Reset if necessary 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1280 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2006-01-03 11:02:02 +00:00
borg-0300
4f18f24d81 small change 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1278 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2006-01-02 22:30:40 +00:00
borg-0300
c652527620 YaCy removes now the old HTCACHE data 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1277 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2006-01-02 22:24:19 +00:00
borg-0300
69f65210e2 ".yacy" has its own directory; 
glad new year :)

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1275 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2006-01-01 13:07:57 +00:00
borg-0300
b95c5d5781 BUGFIX for URLs how "/../" ...; 
new port handling;

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1271 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2005-12-30 12:58:36 +00:00
borg-0300
b7f9adc2c9 new filters added 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1231 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2005-12-18 01:15:25 +00:00
orbiter
83a34b838d * added Object allocation monitor on performanceMemory page 
* added some final statements
* changed shutdown sequence order

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1211 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2005-12-14 13:04:43 +00:00
orbiter
4ff3d219e8 increased delay for cacheScan start and slowed down scan process 
to provide more time to other tasks

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1210 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2005-12-13 21:15:52 +00:00
orbiter
0c762daf4b better startup failure handling 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1205 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2005-12-12 23:59:58 +00:00