Marown/yacy_search_server
1
0
Fork 0
mirror of https://github.com/yacy/yacy_search_server.git synced 2024-09-21 00:00:13 +02:00
Code Issues Actions Packages Projects Releases Wiki Activity
3,297 Commits 1 Branch 20 Tags 836 MiB
7af188ff9a
Commit Graph

2146 Commits

Author SHA1 Message Date
orbiter
7af188ff9a fix for http://www.yacy-forum.de/viewtopic.php?p=33089#33089 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3491 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2007-03-19 11:59:29 +00:00
orbiter
5bbf010107 removed synchronization of size() method from numerous classes to avoid thread locking 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3490 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2007-03-18 19:45:23 +00:00
orbiter
6b9eea3932 - removed differentiation between longTitle and shortTitle; this cannot be used for search results, 
and it is difficult to get both types from all document types
- added some author parsing

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3489 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2007-03-18 12:33:19 +00:00
orbiter
a738b57b31 added author tag to indexing content 
enhanced composition of title tag
TODO: insert author information for external parsers

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3488 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2007-03-17 01:18:34 +00:00
orbiter
6be57983a8 another update to the crawl balancer 
can now alternate between top and bottom of the crawl stack

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3487 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2007-03-16 16:54:54 +00:00
orbiter
91cdc1493f removed query to NAT or responder in case that no other peer is there. 
this is not needed any more, there are enough peers

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3486 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2007-03-16 15:21:24 +00:00
orbiter
4783a30910 - fixed a flush problem in balancer 
- return to idle divisor in RWI RAM cache flush

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3485 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2007-03-16 15:16:26 +00:00
theli
91c2a042a7 *) bugfix for wrong proxy traffic accounting 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3484 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2007-03-16 13:52:48 +00:00
orbiter
861f41e67e redesigned NURL-handling: 
- the general NURL-index for all crawl stack types was splitted into separate indexes for these stacks
- the new NURL-index is managed by the crawl balancer
- the crawl balancer does not need an internal index any more, it is replaced by the NURL-index
- the NURL.Entry was generalized and is now a new class plasmaCrawlEntry
- the new class plasmaCrawlEntry replaces also the preNURL.Entry class, and will also replace the switchboardEntry class in the future
- the new class plasmaCrawlEntry is more accurate for date entries (holds milliseconds) and can contain larger 'name' entries (anchor tag names)
- the EURL object was replaced by a new ZURL object, which is a container for the plasmaCrawlEntry and some tracking information
- the EURL index is now filled with ZURL objects
- a new index delegatedURL holds ZURL objects about plasmaCrawlEntry obects to track which url is handed over to other peers
- redesigned handling of plasmaCrawlEntry - handover, because there is no need any more to convert one entry object into another
- found and fixed numerous bugs in the context of crawl state handling
- fixed a serious bug in kelondroCache which caused that entries could not be removed
- fixed some bugs in online interface and adopted monitor output to new entry objects
- adopted yacy protocol to handle new delegatedURL entries
all old crawl queues will disappear after this update!

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3483 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2007-03-16 13:25:56 +00:00
hydrox
9b5fb3908d *) a peer-message are now created when a blog-comment is written 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3480 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2007-03-15 12:58:17 +00:00
orbiter
581db87237 more debug code for 
http://www.yacy-forum.de/viewtopic.php?p=33009#33009

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3479 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2007-03-14 15:04:06 +00:00
orbiter
81c4cc6bf7 better debugging of balancer failure 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3478 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2007-03-14 12:02:56 +00:00
orbiter
dd06d4cada more logging to better trace bug 
http://www.yacy-forum.de/viewtopic.php?p=33001#33001

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3477 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2007-03-14 09:36:54 +00:00
orbiter
96b79bf86d redesigned remove method in kelondroRowSet 
This should fix also numerous bugs like
http://www.yacy-forum.de/viewtopic.php?p=31077#31077
(java.lang.ArrayIndexOutOfBoundsException in kelondroRowCollection.removeShift)

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3476 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2007-03-14 08:55:05 +00:00
orbiter
9f929b5438 better snippet handling in case of snippet load fail 
see also http://www.yacy-forum.de/viewtopic.php?p=31096#31096

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3475 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2007-03-13 22:18:36 +00:00
auron_x
d451ad48d3 *) improved peerloadgraphic: 
- unnecessary (0 %) pieces are removed
 - percent-values of each thread displayed in legend

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3474 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2007-03-12 19:08:17 +00:00
orbiter
a5d668c0c6 added speed-buttons for easy performance setting 
appears in crawl start and on indexing monitor page

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3473 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2007-03-12 16:24:28 +00:00
orbiter
5b0a84ce09 fix for synchronization deadlock with flushMissNameCache. 
see also: http://www.yacy-forum.de/viewtopic.php?p=32939#32939

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3472 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2007-03-12 09:06:57 +00:00
karlchenofhell
e2ac5f62bd - Code hübscher machen [von NNs TODO] 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3471 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2007-03-11 19:53:14 +00:00
allo
f04097c3dd integrated tor-patch for crawling, if yacyDebugMode is set. 
(replaces: http://yacy.deruwe.de/overlay/net-misc/yacy-tor/files/disable_dns_checks-svn3132.patch)


git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3470 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2007-03-11 18:43:11 +00:00
auron_x
22fe14f292 *) first version of Peerload-graphic 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3469 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2007-03-11 17:04:11 +00:00
orbiter
432d7d4e9c better catch 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3468 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2007-03-10 23:38:08 +00:00
orbiter
8f7e8b6ee2 auto-delete for not-fixable db error in crawl stacker. 
see also http://www.yacy-forum.de/viewtopic.php?p=32906#32906

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3467 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2007-03-10 23:31:36 +00:00
orbiter
7a52b07fcc better memory protection during freemen cycle 
see also http://www.yacy-forum.de/viewtopic.php?p=32903#32903

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3466 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2007-03-10 23:22:37 +00:00
orbiter
6faa262259 fix for NURL-fix 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3465 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2007-03-09 14:30:53 +00:00
orbiter
909d7a8ae9 fixed wrong implemented row iterator in kelomdroFlexSplitTables 
this has no effect, until now this iterator was only used on
the Index Administration page.

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3464 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2007-03-09 13:55:26 +00:00
orbiter
a1fb8358b2 lets make a well-formed http link so that other crawlers don't have a problem to follow this link :-) 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3463 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2007-03-09 12:35:54 +00:00
orbiter
4edb70f68b added yacybot info-page from Roland 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3462 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2007-03-09 12:26:31 +00:00
orbiter
3ef77d2030 fix for http://www.yacy-forum.de/viewtopic.php?p=29878#29878 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3461 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2007-03-09 12:14:25 +00:00
orbiter
3bb3df3fc0 fix for http://www.yacy-forum.de/viewtopic.php?p=32298#32298 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3460 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2007-03-09 12:03:53 +00:00
orbiter
243a2f831b fixed problem with not found NURL-hashes 
The cause for this problem could still not be found, but the effect
is handled much better. The NURL-pop will continue automatically until
it found a hash that can be found.

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3458 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2007-03-09 11:07:20 +00:00
orbiter
6ad39bae1e fixed shutdown problem 
this fixes the 'inconsistency' messages during start-up

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3457 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2007-03-09 08:48:47 +00:00
orbiter
38b93f8cb8 bugfix for my last commit: 
iterator did not consider secondary start point in case of rotation

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3456 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2007-03-08 22:07:17 +00:00
karlchenofhell
264a82eec8 - fix for http://www.yacy-forum.de/viewtopic.php?t=3657 
- fix for http://www.yacy-forum.de/viewtopic.php?p=32758#32758
- Diff takes any objects now, not only strings

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3455 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2007-03-08 22:04:15 +00:00
orbiter
d755a8026d - better OOM protection 
- better memory allocation for FlexTable indexes
- splitting between static index and dynamic index (only the dynamic part must grow)
- to enable a merge-iteration of new splittet index, a huge number of classes needed to be adopted for new iterator classes
- added new iterator classes that support cloneable iterators
- adopted all iterator classes to implement cloneable itarators

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3453 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2007-03-08 16:15:40 +00:00
orbiter
23338d2070 small fix for RAM computation 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3447 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2007-03-07 23:55:52 +00:00
orbiter
33f97cff7a changed startup initialization sequence slightly 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3446 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2007-03-07 23:24:16 +00:00
orbiter
4e8eb1dbe3 some minor changes here and there 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3441 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2007-03-07 14:22:10 +00:00
karlchenofhell
03c5906ae7 - minor bugfixes for url-fetcher & http://www.yacy-forum.de/viewtopic.php?t=3646 
- PerformanceMemory_p.html is valid XHTML again

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3440 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2007-03-07 11:50:03 +00:00
orbiter
3499a364ef a little bit better memory protection 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3439 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2007-03-07 09:38:14 +00:00
orbiter
313f6a7680 fix for http://www.yacy-forum.de/viewtopic.php?p=31553#31553 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3438 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2007-03-07 09:26:01 +00:00
orbiter
958ebea5c5 fix for http://www.yacy-forum.de/viewtopic.php?p=32470#32470 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3437 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2007-03-07 09:08:13 +00:00
orbiter
5d5e6ebfcc fix for http://www.yacy-forum.de/viewtopic.php?p=32631#32631 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3436 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2007-03-07 08:54:07 +00:00
orbiter
1cba31de43 redesigned ram organization for database caches 
- each cache can now allocate as much memory as is available
- no more fixed limits
- replaced old performance memory monitor by new one
- added supervision methods as static functions into the classes that provide cache functionality
- steering of ram allocation is done with two simple limits that are ram availability-relative


git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3434 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2007-03-06 22:43:32 +00:00
theli
26450a1d9a *) avoid nullpointerException on seed.getAddress() (reported by netbude) 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3431 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2007-03-06 16:11:36 +00:00
orbiter
db235f2d61 added some memory protection in collection index multiple merge 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3429 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2007-03-04 22:54:04 +00:00
theli
c72605ecab *) adding a function to determine if a given URL is bookmarkt 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3428 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2007-03-03 11:57:49 +00:00
theli
bd03c6b874 *) bugfix in bookmarksDB: 
- NullpointerException when trying to get an unknown bookmark
   - bookmarks can either start with http or https

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3427 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2007-03-03 11:56:46 +00:00
orbiter
b466baa574 added some memory protection 
too large collection arrays are now avoided. By default, the biggest
collection index is 7. larger collections are dumped into a commons
directory, but cannot yet be used. Bevore doing a dump, the collection
is splittet into a part which has only root-references, and stored back
to the collection; the remaining part goes to commons

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3426 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2007-03-03 00:55:51 +00:00
low012
ce360ef43e *) no more HTML in plasmaCrawlProfile.java anymore 
*) <br> will not be displayed in items in Auto Filter Content on WatchCrawler_p.html anymore
*) removed unnecessary replaceHTML()


git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3425 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2007-03-02 21:09:28 +00:00