Commit Graph

2174 Commits

Author SHA1 Message Date
orbiter
ce9dd3e76d some work in the index construction zone (no effect yet)
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2275 6c8d7289-2bf4-0310-a012-ef5d649a1542
2006-07-03 15:14:54 +00:00
theli
fe617d7e54 *) adding function to return the protocol type of a ssl connection
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2274 6c8d7289-2bf4-0310-a012-ef5d649a1542
2006-07-03 14:16:46 +00:00
orbiter
675d1081a4 code clean-up: switched off unused variables
(bitte solche Variablen die nicht benutzt werden vermeiden.
Ich habe hier ein strafferes Warning an, da bekommt man die
immer angezeigt)

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2273 6c8d7289-2bf4-0310-a012-ef5d649a1542
2006-07-03 13:39:08 +00:00
orbiter
018b3e0832 added pause option to server threads.
The pause is started by calling intermission(Long.MAX_VALUE)
and can be stopped by calling intermission(0)

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2272 6c8d7289-2bf4-0310-a012-ef5d649a1542
2006-07-03 13:20:14 +00:00
orbiter
e1a52bea22 added a class stub for the new database structure:
a reverse word index based on a a collection index,
which is an index for a set of array files containing
row collections.

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2271 6c8d7289-2bf4-0310-a012-ef5d649a1542
2006-07-02 22:24:13 +00:00
orbiter
3b69b35bf2 added pre-load of node cache entries to kelondroRecords
this gives the kelondroTree data structure a similar start-up
behaviour like the kelondroFlexTable: the cache is filled with
routing data in such a way that is more performant than
reading node records during normal operation.
The pre-load phase stops automatically after a time-out of 500 milliseconds
of if the cache is full.

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2270 6c8d7289-2bf4-0310-a012-ef5d649a1542
2006-07-02 22:20:55 +00:00
theli
31b5d8980e *) Bugfix for basic config redirection bug
see: http://www.yacy-forum.de/viewtopic.php?p=23503

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2269 6c8d7289-2bf4-0310-a012-ef5d649a1542
2006-07-02 13:03:01 +00:00
orbiter
85d575e928 enhancements to kelondroRow and kelondroColumn
these are changes towards a better indexURLEntry implementation
which are needed for the new database structures

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2268 6c8d7289-2bf4-0310-a012-ef5d649a1542
2006-07-02 01:26:06 +00:00
orbiter
ab1ed053f5 another small correction
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2267 6c8d7289-2bf4-0310-a012-ef5d649a1542
2006-07-02 00:57:20 +00:00
orbiter
b92561fb67 removed unused code
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2266 6c8d7289-2bf4-0310-a012-ef5d649a1542
2006-07-02 00:44:38 +00:00
orbiter
eadbd56fc5 small adjustment to last commit
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2265 6c8d7289-2bf4-0310-a012-ef5d649a1542
2006-07-02 00:39:23 +00:00
orbiter
e9765ac4e6 introduced bulk read for node iterator in kelondroRecords
this speeds up the iterator by factor 2

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2264 6c8d7289-2bf4-0310-a012-ef5d649a1542
2006-07-02 00:31:20 +00:00
orbiter
6643da3fbd bugfix for http://www.yacy-forum.de/viewtopic.php?p=23463#23463
(affected URL DB Cleaner)

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2263 6c8d7289-2bf4-0310-a012-ef5d649a1542
2006-07-01 09:51:00 +00:00
orbiter
866d53ed70 fix for DNS block bug
see http://www.yacy-forum.de/viewtopic.php?p=23458#23458

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2262 6c8d7289-2bf4-0310-a012-ef5d649a1542
2006-06-30 22:39:33 +00:00
orbiter
6af70febef - added kelondroTree index option to kelondroFlexTable
- automatic generation of index file when index is too large for RAM


git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2261 6c8d7289-2bf4-0310-a012-ef5d649a1542
2006-06-30 12:54:19 +00:00
orbiter
dd2865178a major bugfix (searched a whole week for the bug) for
the kelondroRowBuffer, which has effect mostly to the
kelondroFlexTable but also to all other database functions

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2260 6c8d7289-2bf4-0310-a012-ef5d649a1542
2006-06-30 08:52:39 +00:00
orbiter
f9b9d085c4 just changed testing code
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2259 6c8d7289-2bf4-0310-a012-ef5d649a1542
2006-06-29 23:01:42 +00:00
rramthun
068d82c4c2 Important update of YPStats
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2258 6c8d7289-2bf4-0310-a012-ef5d649a1542
2006-06-29 20:30:50 +00:00
theli
b594ee9a5a *) Adding possibility to configure if the http proxy should send the
X-forwarded-for header (requested by TeeSee)
   See: http://www.yacy-forum.de/viewtopic.php?t=2577

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2257 6c8d7289-2bf4-0310-a012-ef5d649a1542
2006-06-29 16:01:03 +00:00
orbiter
ef84fc4956 added IOException to size() and row()
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2256 6c8d7289-2bf4-0310-a012-ef5d649a1542
2006-06-29 07:56:27 +00:00
theli
7646849241 *) implementation of size() for mysql
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2255 6c8d7289-2bf4-0310-a012-ef5d649a1542
2006-06-29 05:36:37 +00:00
orbiter
84dfd76a6a kelondroFlex bugfix
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2254 6c8d7289-2bf4-0310-a012-ef5d649a1542
2006-06-28 16:14:47 +00:00
rramthun
0e9192a622 Updated YPStats_p.html
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2253 6c8d7289-2bf4-0310-a012-ef5d649a1542
2006-06-28 09:51:38 +00:00
hydrox
53077f5835 *)fixed paths to yacy.logging
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2252 6c8d7289-2bf4-0310-a012-ef5d649a1542
2006-06-28 09:04:53 +00:00
hydrox
8ba8e2b7d9 *) added cache for blacklists urlhashs recieved by DHT. DHT does not request URLs listed in this cache.
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2251 6c8d7289-2bf4-0310-a012-ef5d649a1542
2006-06-28 08:51:34 +00:00
hydrox
9f55973fe2 *) added some OpenDocument extentions to httpd.mime
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2250 6c8d7289-2bf4-0310-a012-ef5d649a1542
2006-06-28 08:08:20 +00:00
rramthun
db2405533a New YPstats possibility.
Users can enter the details of their system configuration and we are able do work-out profiles for special uses and hardware-configurations.
Default DB running on yacy-forum.de, but URL can be changed. 
User can see all data, which will be submitted.

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2249 6c8d7289-2bf4-0310-a012-ef5d649a1542
2006-06-27 15:45:38 +00:00
hermens
53cbcc6d6e Implement emergency break in index receive when the limit of the ramCache is exceeded by more than cacheLimit
See: http://www.yacy-forum.de/viewtopic.php?p=22911#22911



git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2248 6c8d7289-2bf4-0310-a012-ef5d649a1542
2006-06-27 11:14:30 +00:00
orbiter
e40987ecab removed default memory reservation for testing
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2247 6c8d7289-2bf4-0310-a012-ef5d649a1542
2006-06-27 09:16:07 +00:00
allo
6e9efca5f5 old version, but not broken
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2246 6c8d7289-2bf4-0310-a012-ef5d649a1542
2006-06-26 20:03:51 +00:00
orbiter
4cc6e6551f bugfix
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2245 6c8d7289-2bf4-0310-a012-ef5d649a1542
2006-06-26 09:52:23 +00:00
orbiter
66964dc015 removed high/med/low from kelondroRecords cache control.
this was done because testing showed that cache-delete operations
slowed down record access most, even more that actual IO operations.
Cache-delete operations appeared when entries were shifted from low-priority
positions to high-priority positions. During a fill of x entries to a database,
x/2 delete situation happen which caused two or more delete operations.
removing the cache control means that these delete operations are not
necessary any more, but it is more difficult to decide which cache elements
shall be removed in case that the cache is full. There is not yet a stable
solution for this case, but the advantage of a faster cache is more important
that the flush problem.

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2244 6c8d7289-2bf4-0310-a012-ef5d649a1542
2006-06-25 10:31:38 +00:00
allo
6866bc2758 be quiet!
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2243 6c8d7289-2bf4-0310-a012-ef5d649a1542
2006-06-24 17:40:55 +00:00
borg-0300
4c6083b264 network picture;
back to old version

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2242 6c8d7289-2bf4-0310-a012-ef5d649a1542
2006-06-24 02:52:24 +00:00
borg-0300
955915385a network picture;
small changes;

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2241 6c8d7289-2bf4-0310-a012-ef5d649a1542
2006-06-23 15:37:59 +00:00
borg-0300
027fa8ab1c network picture;
bigger; 
more dot steps; 
small other;

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2240 6c8d7289-2bf4-0310-a012-ef5d649a1542
2006-06-23 13:53:29 +00:00
orbiter
41c4641612 added some profiling to kelondro caching classes
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2239 6c8d7289-2bf4-0310-a012-ef5d649a1542
2006-06-23 12:49:42 +00:00
orbiter
dd560e4b2f finetuning
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2238 6c8d7289-2bf4-0310-a012-ef5d649a1542
2006-06-22 23:11:40 +00:00
orbiter
a5661de880 added debug line
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2237 6c8d7289-2bf4-0310-a012-ef5d649a1542
2006-06-22 15:45:33 +00:00
orbiter
5b1d77cd4b some enhancements to caching
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2236 6c8d7289-2bf4-0310-a012-ef5d649a1542
2006-06-22 15:42:09 +00:00
hydrox
ad03715b64 *) added fill and read functions with benchmark output every 10000 operations to dbtest.java
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2235 6c8d7289-2bf4-0310-a012-ef5d649a1542
2006-06-22 08:37:48 +00:00
theli
b20496e42b *) make DHT DoS check configurable (requested by KoH)
- check can be disabled via property indexDistribution.dhtReceiptLimitEnabled
   - upper bound can be configured via indexDistribution.dhtReceiptLimit

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2234 6c8d7289-2bf4-0310-a012-ef5d649a1542
2006-06-21 19:28:42 +00:00
orbiter
650c7e9e55 some enhancements to caching
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2233 6c8d7289-2bf4-0310-a012-ef5d649a1542
2006-06-21 16:05:31 +00:00
hermens
d4645062bc Correct usage of vhost in wget/wput requests:
- yacyClient: don't use own .yacyh domain in requests, instead use .yacyh domain of target peer for everything but ranking distribution
- natLib: use full hostname instead of just SLD.TLD



git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2232 6c8d7289-2bf4-0310-a012-ef5d649a1542
2006-06-21 14:25:27 +00:00
orbiter
bde748f68f - integrated buffer in RowCollectionSet which speeds up the new
cache by factor 3. (but the new cache was 5 times slower than the
old cache...)
- removed old unused code


git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2231 6c8d7289-2bf4-0310-a012-ef5d649a1542
2006-06-21 14:16:34 +00:00
theli
fa9712151a *) dbtest.java: bugfix for mysql test. data was not written correctly
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2230 6c8d7289-2bf4-0310-a012-ef5d649a1542
2006-06-21 13:01:06 +00:00
theli
f55d3d0d79 *) better error message if jdbc driver could not be loaded
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2229 6c8d7289-2bf4-0310-a012-ef5d649a1542
2006-06-21 12:14:04 +00:00
theli
07aa779e6d *) dbtest: adding read and write methods for mysql
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2228 6c8d7289-2bf4-0310-a012-ef5d649a1542
2006-06-21 11:44:38 +00:00
allo
032b3fdd94 allow Creation of gettext only with password, because it causes high cpu-usage.
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2227 6c8d7289-2bf4-0310-a012-ef5d649a1542
2006-06-21 10:39:25 +00:00
orbiter
d083b15c4b tests/fixes
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2226 6c8d7289-2bf4-0310-a012-ef5d649a1542
2006-06-20 23:47:51 +00:00