3 Commits

Author	SHA1	Message	Date
orbiter	1e6d12f146	Major update to BLOB data structures: ... - introduced a new BLOB file format: kelondroBLOBHeap. This is a flat file with an index in RAM. very similar to the eco-tables, but with flexible value sizes. It will replace the kelondroBLOBTree, which is based on a kelondroTree, a file-AVL-based index data structure. - the HTCACHE header file was replaced by the new blob heap file structure - the robots.txt file was replaced by the new blob heap file structure - the robots parser was enhanced (bugfixing for double-loading of the same robots.txt) - other BLOB-dependent data structures were prepared to use also the new BLOB heap - fixed a bug in the snippet fetch process: the file header was not written to the header index There should now be less IO during snippet fetch and during crawling git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4978 6c8d7289-2bf4-0310-a012-ef5d649a1542	2008-07-10 00:47:37 +00:00
orbiter	81f75f5056	- removed unnecessary classes (these objects are much easier to handle using generics) ... - generalized BLOB referencing. This is the preparation to use another BLOB class, the kelondroHeap git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4977 6c8d7289-2bf4-0310-a012-ef5d649a1542	2008-07-07 23:52:53 +00:00
orbiter	3330181aa0	refactoring: ... find a better way to store BLOBs; generalize current BLOG data structure (kelondroDyn) and prepare it to replace it with something better. The best candidate is the kelondroHeap, which will become the kelondroBLOBHeap; removed also some never-used classes git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4902 6c8d7289-2bf4-0310-a012-ef5d649a1542	2008-06-07 23:12:24 +00:00