Commit Graph

8 Commits

Author SHA1 Message Date
theli
40dd6ec4fd *) experimental restructuring of db import function
- trying to reduce IO load by avoiding  unnecessary db access
   - trying to presort url list

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1671 6c8d7289-2bf4-0310-a012-ef5d649a1542
2006-02-16 13:07:01 +00:00
orbiter
7eb10675b3 re-organization of index management
this was done to be prepared for new storage algorithms


git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1635 6c8d7289-2bf4-0310-a012-ef5d649a1542
2006-02-14 00:12:07 +00:00
orbiter
1e4578aab6 VERY EXPERIMENTAL removal of index ram cache flushing thread.
The cache will fill up and flushed explicitely when it is full.
This shall remove double-access of assortments (indexing and flush)
during indexing process. Hopefully this should reduce IO.
The main idea is: the cache shall mainly be flushed by DHT transfer, and
only indexes that shall be hosted by the own peer are flushed to the
assortments. This needs further work.

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1617 6c8d7289-2bf4-0310-a012-ef5d649a1542
2006-02-11 23:19:01 +00:00
orbiter
be77fe1a88 code clean-up
@Martin: bitte schaue mal warum die Variablenzuweisung
         in plasmaCrawlNURLImporter war. So wie sie waren, waren sie überflüssig.
         Das hattest du dir bestimmt nicht so gedacht.
         Sollten es ggf. globale Variablen sein?

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1529 6c8d7289-2bf4-0310-a012-ef5d649a1542
2006-02-04 15:25:48 +00:00
theli
50d85657b8 *) new import function for IndexImport_p.html
- can be used to import the crawling queue (noticeUrlDB + stacks)
   

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1518 6c8d7289-2bf4-0310-a012-ef5d649a1542
2006-02-02 16:46:58 +00:00
theli
442807cb29 *) Bugfix for last commit
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1506 6c8d7289-2bf4-0310-a012-ef5d649a1542
2006-01-31 15:26:11 +00:00
theli
22fd1ca9aa *) minor changes
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1505 6c8d7289-2bf4-0310-a012-ef5d649a1542
2006-01-31 12:39:10 +00:00
theli
6a99304b2b *) Redesign of db import functionality
- restructuring to allow different import tasks to be controlled via one gui 
   - adding possibility to import a single assortment file
   - adding possibility to set the cache size that should be used

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1504 6c8d7289-2bf4-0310-a012-ef5d649a1542
2006-01-31 12:30:24 +00:00