Marown/yacy_search_server
1
0
Fork 0
mirror of https://github.com/yacy/yacy_search_server.git synced 2024-09-21 00:00:13 +02:00
Code Issues Actions Packages Projects Releases Wiki Activity
2,176 Commits 1 Branch 20 Tags 836 MiB
671fd9a5c9
Commit Graph

1422 Commits

Author SHA1 Message Date
orbiter
671fd9a5c9 work towards new indexing database structure 
(no effect on current functionality yet)

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2277 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2006-07-04 14:47:27 +00:00
orbiter
92f4cb4d73 added option to configure the start-up delay time for kelondro database files. 
the start-up delay is used to pre-load the database node cache

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2276 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2006-07-03 23:57:33 +00:00
orbiter
ce9dd3e76d some work in the index construction zone (no effect yet) 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2275 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2006-07-03 15:14:54 +00:00
theli
fe617d7e54 *) adding function to return the protocol type of a ssl connection 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2274 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2006-07-03 14:16:46 +00:00
orbiter
018b3e0832 added pause option to server threads. 
The pause is started by calling intermission(Long.MAX_VALUE)
and can be stopped by calling intermission(0)

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2272 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2006-07-03 13:20:14 +00:00
orbiter
e1a52bea22 added a class stub for the new database structure: 
a reverse word index based on a a collection index,
which is an index for a set of array files containing
row collections.

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2271 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2006-07-02 22:24:13 +00:00
orbiter
3b69b35bf2 added pre-load of node cache entries to kelondroRecords 
this gives the kelondroTree data structure a similar start-up
behaviour like the kelondroFlexTable: the cache is filled with
routing data in such a way that is more performant than
reading node records during normal operation.
The pre-load phase stops automatically after a time-out of 500 milliseconds
of if the cache is full.

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2270 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2006-07-02 22:20:55 +00:00
orbiter
85d575e928 enhancements to kelondroRow and kelondroColumn 
these are changes towards a better indexURLEntry implementation
which are needed for the new database structures

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2268 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2006-07-02 01:26:06 +00:00
orbiter
ab1ed053f5 another small correction 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2267 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2006-07-02 00:57:20 +00:00
orbiter
b92561fb67 removed unused code 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2266 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2006-07-02 00:44:38 +00:00
orbiter
eadbd56fc5 small adjustment to last commit 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2265 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2006-07-02 00:39:23 +00:00
orbiter
e9765ac4e6 introduced bulk read for node iterator in kelondroRecords 
this speeds up the iterator by factor 2

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2264 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2006-07-02 00:31:20 +00:00
orbiter
6643da3fbd bugfix for http://www.yacy-forum.de/viewtopic.php?p=23463#23463 
(affected URL DB Cleaner)

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2263 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2006-07-01 09:51:00 +00:00
orbiter
866d53ed70 fix for DNS block bug 
see http://www.yacy-forum.de/viewtopic.php?p=23458#23458

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2262 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2006-06-30 22:39:33 +00:00
orbiter
6af70febef - added kelondroTree index option to kelondroFlexTable 
- automatic generation of index file when index is too large for RAM


git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2261 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2006-06-30 12:54:19 +00:00
orbiter
dd2865178a major bugfix (searched a whole week for the bug) for 
the kelondroRowBuffer, which has effect mostly to the
kelondroFlexTable but also to all other database functions

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2260 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2006-06-30 08:52:39 +00:00
orbiter
f9b9d085c4 just changed testing code 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2259 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2006-06-29 23:01:42 +00:00
theli
b594ee9a5a *) Adding possibility to configure if the http proxy should send the 
X-forwarded-for header (requested by TeeSee)
   See: http://www.yacy-forum.de/viewtopic.php?t=2577

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2257 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2006-06-29 16:01:03 +00:00
orbiter
ef84fc4956 added IOException to size() and row() 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2256 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2006-06-29 07:56:27 +00:00
orbiter
84dfd76a6a kelondroFlex bugfix 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2254 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2006-06-28 16:14:47 +00:00
hydrox
8ba8e2b7d9 *) added cache for blacklists urlhashs recieved by DHT. DHT does not request URLs listed in this cache. 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2251 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2006-06-28 08:51:34 +00:00
hermens
53cbcc6d6e Implement emergency break in index receive when the limit of the ramCache is exceeded by more than cacheLimit 
See: http://www.yacy-forum.de/viewtopic.php?p=22911#22911



git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2248 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2006-06-27 11:14:30 +00:00
orbiter
e40987ecab removed default memory reservation for testing 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2247 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2006-06-27 09:16:07 +00:00
orbiter
4cc6e6551f bugfix 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2245 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2006-06-26 09:52:23 +00:00
orbiter
66964dc015 removed high/med/low from kelondroRecords cache control. 
this was done because testing showed that cache-delete operations
slowed down record access most, even more that actual IO operations.
Cache-delete operations appeared when entries were shifted from low-priority
positions to high-priority positions. During a fill of x entries to a database,
x/2 delete situation happen which caused two or more delete operations.
removing the cache control means that these delete operations are not
necessary any more, but it is more difficult to decide which cache elements
shall be removed in case that the cache is full. There is not yet a stable
solution for this case, but the advantage of a faster cache is more important
that the flush problem.

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2244 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2006-06-25 10:31:38 +00:00
allo
6866bc2758 be quiet! 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2243 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2006-06-24 17:40:55 +00:00
borg-0300
4c6083b264 network picture; 
back to old version

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2242 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2006-06-24 02:52:24 +00:00
borg-0300
955915385a network picture; 
small changes;

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2241 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2006-06-23 15:37:59 +00:00
borg-0300
027fa8ab1c network picture; 
bigger; 
more dot steps; 
small other;

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2240 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2006-06-23 13:53:29 +00:00
orbiter
41c4641612 added some profiling to kelondro caching classes 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2239 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2006-06-23 12:49:42 +00:00
orbiter
dd560e4b2f finetuning 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2238 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2006-06-22 23:11:40 +00:00
orbiter
5b1d77cd4b some enhancements to caching 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2236 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2006-06-22 15:42:09 +00:00
theli
b20496e42b *) make DHT DoS check configurable (requested by KoH) 
- check can be disabled via property indexDistribution.dhtReceiptLimitEnabled
   - upper bound can be configured via indexDistribution.dhtReceiptLimit

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2234 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2006-06-21 19:28:42 +00:00
orbiter
650c7e9e55 some enhancements to caching 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2233 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2006-06-21 16:05:31 +00:00
hermens
d4645062bc Correct usage of vhost in wget/wput requests: 
- yacyClient: don't use own .yacyh domain in requests, instead use .yacyh domain of target peer for everything but ranking distribution
- natLib: use full hostname instead of just SLD.TLD



git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2232 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2006-06-21 14:25:27 +00:00
orbiter
bde748f68f - integrated buffer in RowCollectionSet which speeds up the new 
cache by factor 3. (but the new cache was 5 times slower than the
old cache...)
- removed old unused code


git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2231 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2006-06-21 14:16:34 +00:00
orbiter
d083b15c4b tests/fixes 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2226 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2006-06-20 23:47:51 +00:00
orbiter
ce9a98ee25 experiments with buffer for collection class 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2225 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2006-06-20 23:18:41 +00:00
orbiter
6aa0f6eb0f better abstraction, less synchronization 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2224 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2006-06-20 22:13:17 +00:00
orbiter
1ec07fe351 better structure of the kelondro collection classes 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2223 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2006-06-20 21:41:12 +00:00
allo
c264cda678 support to parse gettext files and to generate updated files, where only the new strings are empty. 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2222 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2006-06-20 18:52:17 +00:00
orbiter
761d3bba54 integrated kelondroRowCollection in kelondroCollectionIndex 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2221 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2006-06-20 14:46:28 +00:00
orbiter
8f0c1a240a added kelondroRowCollection and kelondroRowSet which 
shall replace kelondroCollection

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2220 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2006-06-20 14:17:21 +00:00
theli
ed2cb040d1 *) Bugfix for http connection header validation 
- Connection header was not handled correctly if it contains
     multiple values, e.g. Connection: TE, close 

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2219 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2006-06-20 05:22:55 +00:00
allo
f95c006dc2 generation of empty gettext files from yacy html-files 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2218 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2006-06-19 19:34:00 +00:00
orbiter
4fa3271754 adopted memory calculation to new memory amount used by kelondroCollectionObjectMap 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2215 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2006-06-18 12:16:56 +00:00
allo
0621106ef3 git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2214 6c8d7289-2bf4-0310-a012-ef5d649a1542 2006-06-18 12:15:26 +00:00
orbiter
11951aed41 replaced node cache Object type in kelondroTree: 
kelondroCollectionObjectMap replaces HashMap.
the kelondroCollectionObjectMap uses only 1/4 of the memory of HashMap,
but is slightly slower than the HashMap.

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2213 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2006-06-18 12:04:41 +00:00
orbiter
12af69dd86 cosmetics 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2212 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2006-06-18 11:49:31 +00:00
orbiter
ea72a7c580 added new objectMap (can replace HashMap and should use less memory, but is slower) 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2211 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2006-06-18 11:33:26 +00:00