Marown/yacy_search_server
1
0
Fork 0
mirror of https://github.com/yacy/yacy_search_server.git synced 2024-09-21 00:00:13 +02:00
Code Issues Actions Packages Projects Releases Wiki Activity
1,634 Commits 1 Branch 20 Tags 836 MiB
a840755964
Commit Graph

549 Commits

Author SHA1 Message Date
orbiter
a840755964 moved parts of index transfer logic back to switchboard 
this is needed to merge the dht selection with the indexing thread

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1718 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2006-02-20 23:27:11 +00:00
orbiter
134253a603 fixed bug with cache flush 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1717 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2006-02-20 23:25:48 +00:00
orbiter
c2d863855d different flush limit 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1713 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2006-02-20 22:21:53 +00:00
borg-0300
64441b1f78 ADDED: yacy.badwords list to filter the topwords 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1711 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2006-02-20 17:50:42 +00:00
orbiter
f9063e2040 added some synchronization to avoid that several tasks can trigger a cache flush simultanously 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1708 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2006-02-20 00:06:03 +00:00
orbiter
2c4e4ae6a2 further refactoring of dht selection, transfer and flushing 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1707 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2006-02-19 23:47:45 +00:00
orbiter
73dad68cf1 outsourced thelis DHT flush class into own file 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1706 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2006-02-19 21:54:46 +00:00
allo
aa4b04e3dd reverted last change 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1705 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2006-02-19 19:05:46 +00:00
allo
4b0dae8fcf added a possiblity to get the ranking values for an url. 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1703 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2006-02-19 11:08:06 +00:00
orbiter
85ac7d8386 * moved DHT transfer thread to own class file, needed for further modularization 
* changed status handling
* added forced cache flush when cache has containers with too high number of index entries


git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1702 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2006-02-19 01:36:42 +00:00
orbiter
7df2e6e571 bugfix for last commit 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1700 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2006-02-19 00:35:23 +00:00
orbiter
cd41e9a0eb moved DHT index selection to new object that holds indexes to be send away to other peer. 
This was made to make it possible that RWI selections can be serialized with indexing.
Serialization will be implemented in another step.

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1698 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2006-02-18 19:34:16 +00:00
theli
42a5f56723 *) Bugfix for broken dht thread configuration 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1695 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2006-02-18 14:51:01 +00:00
theli
f95d98142f *) displaying amount of items in the existsIndex caches 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1679 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2006-02-17 11:02:01 +00:00
hydrox
e2af2a3f45 *) it's now possible to run more then one indexDistribution-Thread 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1673 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2006-02-16 15:22:25 +00:00
theli
40dd6ec4fd *) experimental restructuring of db import function 
- trying to reduce IO load by avoiding  unnecessary db access
   - trying to presort url list

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1671 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2006-02-16 13:07:01 +00:00
theli
2da18ab359 *) correcting logging output 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1667 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2006-02-16 10:56:17 +00:00
theli
8ffc6e35ad *) correcting logging output 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1665 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2006-02-16 10:28:54 +00:00
theli
980e986b64 *) Re enabling short cycle for already removed nurl entries 
See: http://www.yacy-forum.de/viewtopic.php?p=17147#17147 

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1660 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2006-02-16 06:59:34 +00:00
hermens
3b6328ad02 *) Consistent use of minCount for index transfer 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1645 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2006-02-14 22:50:13 +00:00
hermens
0b60b9bf51 *) Remove entries from AssortmentCluster before reinserting the rest into the ramCache 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1640 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2006-02-14 13:06:32 +00:00
hydrox
8ab1d6ff4b *) fixed NullPointerException in plasmaWordIndexEntity 
See: http://www.yacy-forum.de/viewtopic.php?t=1921

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1638 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2006-02-14 11:40:36 +00:00
allo
a26574c894 Migration from tagName as key to wordhash(tagName) as key for bookmarkTags.db 
(just deleting the old db, rebuildTags does the rest)

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1637 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2006-02-14 10:00:12 +00:00
orbiter
7eb10675b3 re-organization of index management 
this was done to be prepared for new storage algorithms


git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1635 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2006-02-14 00:12:07 +00:00
orbiter
1e4578aab6 VERY EXPERIMENTAL removal of index ram cache flushing thread. 
The cache will fill up and flushed explicitely when it is full.
This shall remove double-access of assortments (indexing and flush)
during indexing process. Hopefully this should reduce IO.
The main idea is: the cache shall mainly be flushed by DHT transfer, and
only indexes that shall be hosted by the own peer are flushed to the
assortments. This needs further work.

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1617 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2006-02-11 23:19:01 +00:00
hermens
954f02d22e *) Bugfix: Prevent wordIndex.getContainer() from returning and even manipulating 
the containers from the ram cache. Return a new container instead.
*) Speedup flushFromMem by reducing the number of searches in the TreeMap



git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1604 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2006-02-11 03:15:14 +00:00
orbiter
fe39493145 changed default ranking parameters 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1582 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2006-02-09 00:23:54 +00:00
orbiter
365a3fff8e fixings for ranking attributes 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1569 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2006-02-07 23:16:46 +00:00
orbiter
8e55098b74 fixed detailed search 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1562 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2006-02-06 23:13:54 +00:00
orbiter
0cb940a8e5 added detailed search. 
ranking profiles do not work properly yet

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1551 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2006-02-05 23:38:00 +00:00
orbiter
c695928f7c adopted search page to new detailed search (to be commited later) 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1550 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2006-02-05 22:51:37 +00:00
orbiter
45323e7b76 fixed null pointer exception during search 
see http://www.yacy-forum.de/viewtopic.php?p=16429#16429


git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1547 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2006-02-05 14:59:30 +00:00
orbiter
fb7411d7bb re-structuring of ranking application: 
concentration of all ranking attributes in the
plasmaSearchRankingProfile

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1541 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2006-02-05 01:47:51 +00:00
orbiter
d98418390b - introduced rankingProfile Class 
- selection of ranking and timing profiles for each search


git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1539 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2006-02-04 23:51:00 +00:00
orbiter
eab1805bca refactoring: plasmaSearchProfile -> plasmaSearchTimingProfile 
This was made to distiguish this profile from the
(to-be-implemented) plasmaSeachOrderProfile

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1538 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2006-02-04 23:11:31 +00:00
orbiter
6eef848954 re-design of post-ranking process 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1537 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2006-02-04 23:00:31 +00:00
orbiter
be77fe1a88 code clean-up 
@Martin: bitte schaue mal warum die Variablenzuweisung
         in plasmaCrawlNURLImporter war. So wie sie waren, waren sie überflüssig.
         Das hattest du dir bestimmt nicht so gedacht.
         Sollten es ggf. globale Variablen sein?

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1529 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2006-02-04 15:25:48 +00:00
orbiter
0bc2aaeb42 added normalization to search attributes 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1528 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2006-02-04 14:13:54 +00:00
theli
008bcb7fb8 *) simplifying code by moving closeTransferIndexes into final block 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1522 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2006-02-03 13:14:30 +00:00
theli
50d85657b8 *) new import function for IndexImport_p.html 
- can be used to import the crawling queue (noticeUrlDB + stacks)
   

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1518 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2006-02-02 16:46:58 +00:00
theli
214302284e *) undoing last commit because of problems with getUpdateTime 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1514 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2006-02-02 10:05:16 +00:00
theli
408de3beee *) avoiding to search in the treemap two times for the same key 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1513 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2006-02-02 10:03:47 +00:00
borg-0300
139ba4e0c8 Bugfix for getCachePath(URL url) 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1510 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2006-02-01 19:07:49 +00:00
theli
442807cb29 *) Bugfix for last commit 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1506 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2006-01-31 15:26:11 +00:00
theli
22fd1ca9aa *) minor changes 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1505 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2006-01-31 12:39:10 +00:00
theli
6a99304b2b *) Redesign of db import functionality 
- restructuring to allow different import tasks to be controlled via one gui 
   - adding possibility to import a single assortment file
   - adding possibility to set the cache size that should be used

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1504 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2006-01-31 12:30:24 +00:00
orbiter
3834675084 fixed bug that caused wrong behavior of search result preparation 
(second search on same topic resulted in less links)

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1502 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2006-01-31 01:20:28 +00:00
hermens
31c8476b5d plasmaWordIndexCache.getContainer: 
*) Also get entries from cache
*) calculate available remaining time for backend.getContainer correctly



git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1501 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2006-01-31 01:04:23 +00:00
orbiter
3419b3bcdd fix for bug that caused the peer-counter problem. 
See http://www.yacy-forum.de/viewtopic.php?p=16016#16016
The kelondroDyn now uses a generic fill character.
kelondroDyn-Tables containing peer/word/url-hashes must not use '_'
as fill character.

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1498 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2006-01-30 22:16:55 +00:00
hermens
4f43816ec0 *) Fix wrong class cast in indexSize() 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1495 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2006-01-30 17:09:07 +00:00