Marown/yacy_search_server
1
0
Fork 0
mirror of https://github.com/yacy/yacy_search_server.git synced 2024-09-21 00:00:13 +02:00
Code Issues Actions Packages Projects Releases Wiki Activity
5,705 Commits 1 Branch 20 Tags 836 MiB
a704d82280
Commit Graph

834 Commits

Author SHA1 Message Date
orbiter
a704d82280 patch for problem with digest 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6031 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-06-06 16:53:16 +00:00
orbiter
3029ef6eb3 fixed a bug that was recently inserted which caused that no idx and gap files were written. 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6030 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-06-06 16:43:58 +00:00
orbiter
95e8cbd1c3 new fully redesigned balancer and bugfixes regarding lost profile handles and killed crawls 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6025 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-06-06 01:56:31 +00:00
orbiter
42ae40b9f6 some bugfixes to database close() methods 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6023 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-06-04 22:43:46 +00:00
orbiter
9bfd22f65d fix for http://forum.yacy-websuche.de/viewtopic.php?p=15523#p15523 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6020 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-06-04 19:57:25 +00:00
orbiter
cc49aedf12 - fixed problem with remote search NPE 
- more abstraction for search requests

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6015 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-06-03 08:49:54 +00:00
orbiter
c38c852090 modified access method to get index entries out of a array of BLOBs: 
iterate them, then merge; not collect them and merge then.
This should use less memory and may behave better in an environment with many queries.
To ensure that too many queries will not cause total blocking,
a time-out of one second was also added. After the time-out
the index data that was collected so far is returned.

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6013 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-06-02 16:53:45 +00:00
orbiter
a5d481eab1 enhanced navigation 
- fixed too early computation of navigation
- moved navigation rendering to yacysearchtrailer
- added more asserts

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6006 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-06-01 22:45:28 +00:00
orbiter
1c69d9b8b6 more refactoring of the index classes 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5995 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-05-29 14:16:41 +00:00
orbiter
88426912ad more refactoring to make the segment object easier to use and to be prepared to integrate author navigation 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5992 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-05-29 10:03:35 +00:00
lotus
d813fd26ed reset sent/received counters on index delete 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5991 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-05-28 15:49:42 +00:00
orbiter
99bf0b8e41 refactoring of plasmaWordIndex: 
divided that class into three parts:
- the peers object is now hosted by the plasmaSwitchboard
- the crawler elements are now in a new class, crawler.CrawlerSwitchboard
- the index elements are core of the new segment data structure, which is a bundle of different indexes for the full text and (in the future) navigation indexes and the metadata store. The new class is now in kelondro.text.Segment

The refactoring is inspired by the roadmap to create index segments, the option to host different indexes on one peer.

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5990 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-05-28 14:26:05 +00:00
orbiter
876746602d catch problems of file hash computation, see also: 
http://forum.yacy-websuche.de/viewtopic.php?p=15245#p15245

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5989 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-05-28 10:08:36 +00:00
orbiter
fec6f9054f some refactoring of search methods 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5988 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-05-27 23:51:34 +00:00
orbiter
3d4b826ca5 migration of all databases that use the deprecated BLOBTree format into the BLOBHeap format. Old databases are migrated automatically. 
This removes the last very IO-intensive data structures which were still used for Wiki, Blog and Bookmarks. Old database files will still remain in the DATA subdirectory but can be deleted manually if no major bugs appear during migration. There is no need for any user action, all migration is done automatically.

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5986 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-05-27 15:04:04 +00:00
orbiter
26a46b5521 increased default maximum file size for database files to 2GB 
Other file sizes can now be configured with the attributes
filesize.max.win and filesize.max.other
the default maximum file size for non-windows OS is now 32GB

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5974 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-05-25 06:59:21 +00:00
orbiter
addecdb18c simplified code, removed one unused method in all implementing classes 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5972 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-05-21 23:53:01 +00:00
borg-0300
47fce9020c small change (Orbiter's wish) 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5971 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-05-21 17:51:52 +00:00
borg-0300
e07b14e5d7 finally a working fix for 5960 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5970 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-05-21 16:07:04 +00:00
borg-0300
3ebb904d2c fix for 5960, http://forum.yacy-websuche.de/viewtopic.php?f=6&t=2119 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5969 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-05-21 11:47:57 +00:00
orbiter
e005cfea37 fix for bug in -incell option of URLAnalysis 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5967 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-05-21 06:57:03 +00:00
orbiter
a7e392f31b The collection index will not be supported any more. 
Existing indexes based on the old index collections must be migrated with YaCy 0.8
- removed index collection classes and all migration tools
- added a 'incell' reference collection feature in URL analysis


git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5966 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-05-20 14:51:26 +00:00
orbiter
a2f48863fc - added prototype for navigation index 
- refactoring of word index prototype
(no functional changes so far)

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5965 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-05-20 09:00:24 +00:00
orbiter
b7457d3807 patch for http://forum.yacy-websuche.de/viewtopic.php?p=14720#p14720 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5960 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-05-17 21:44:02 +00:00
orbiter
f133d6065c fix for http://forum.yacy-websuche.de/viewtopic.php?p=14955#p14955 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5958 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-05-17 18:28:33 +00:00
orbiter
ad9762746d no exception in case of uniq() time-out, see also 
http://forum.yacy-websuche.de/viewtopic.php?p=13177#p13177

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5955 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-05-15 23:07:10 +00:00
orbiter
f2e4d156e8 removed debug messages 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5950 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-05-12 22:03:33 +00:00
orbiter
c1e5fad9a7 fix for http://forum.yacy-websuche.de/viewtopic.php?p=14767#p14767 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5944 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-05-10 20:50:46 +00:00
orbiter
8ee3a94e82 fix for non-caching of sitehash, see http://forum.yacy-websuche.de/viewtopic.php?p=14440#p14440 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5942 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-05-10 11:44:17 +00:00
borg-0300
21930d05ed fix for [B@... 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5941 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-05-10 10:54:06 +00:00
orbiter
b6ba387e01 fix for http://forum.yacy-websuche.de/viewtopic.php?p=14751#p14751 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5940 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-05-10 10:46:26 +00:00
low012
d164b42604 *) cosmetics 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5934 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-05-08 19:26:36 +00:00
orbiter
5fb77116c6 added a submenu to index administration to import a wikimedia dump (i.e. a dump from wikipedia) into the YaCy index: see 
http://localhost:8080/IndexImportWikimedia_p.html

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5930 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-05-08 07:54:10 +00:00
hermens
df733af4fa Try not to loose content from ram during IndexCell.delete by moving ram.delete after the dangerous operations on the array (array.get and array.delete) 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5929 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-05-06 12:53:17 +00:00
hermens
ac72005f2f Let IndexCell.remove remove entries from the ram portion of the DB as well. 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5928 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-05-06 12:32:34 +00:00
orbiter
8ba7ff5353 a fix and another speed enhancement for the RWI cache 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5927 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-05-05 22:40:40 +00:00
orbiter
05f077e85f added stack trace output to solve problem in 
http://forum.yacy-websuche.de/viewtopic.php?f=6&t=2076&hilit=&p=14612#p14612

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5926 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-05-05 20:24:20 +00:00
orbiter
71a4cadf31 better and more performant synchronization in SimpleARC, the caching object for word hashes. Speeds up indexing. 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5925 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-05-05 20:19:51 +00:00
orbiter
e6773cbb33 better handling of RWI cache for concurrency and less overhead when writing new entries -> even more indexing speed 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5924 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-05-05 20:08:23 +00:00
orbiter
c097531e3d added a catch Exception to all thread to check if any of them silently dies without any other notification 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5922 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-05-05 06:31:35 +00:00
orbiter
083533e5ec fix for bugs in IODispatcher 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5921 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-05-04 21:37:59 +00:00
orbiter
21fbca0410 better scaling of HEAP dump writer for small memory configurations; 
should prevent OOMs during cache dumps

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5920 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-05-04 08:29:44 +00:00
orbiter
6e0b57284d better care for states of the IODispatcher 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5919 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-05-03 22:54:47 +00:00
orbiter
1db9cdd4e4 fixed bug in writing of robots.txt entries in case that host names exceeded 64 characters and some other problems 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5918 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-05-03 19:35:10 +00:00
orbiter
057ce14c8e more fixes (character encoding, parser exceptions, http client failure, blob writing) 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5914 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-05-02 07:43:03 +00:00
orbiter
d2ac0aa682 - fixed possible bugs in Stack (may affect Crawler reset) and RandomAccess handling 
- increased default memory size to 180MB
- fixed possible bug in http client reset (there was a deadlock)
- bug in BOBHeap marked, but not solved, cause is still unknown.

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5912 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-05-02 01:40:03 +00:00
orbiter
8d6212233b fix for IODispatcher 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5896 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-04-28 07:24:28 +00:00
orbiter
07f09742bb set of small fixes and comments 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5893 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-04-27 15:29:50 +00:00
orbiter
9e4db75aac reduced internal logging and reduced memory that internal logging can use 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5867 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-04-24 12:09:04 +00:00
orbiter
c10c257255 attempt to fix a deadlock situation where the IODispatcher did not work. 
I suspect the dispatcher thread has crashed and queues filled so no indexing process was able to write data.
This fix tries to heal the problem, but I am unsure if it helps. To get a better view of the problem, some more log outputs had been inserted.
Added also a new attribut indexer.threads to get a control over the number of default threads for the indexer (default is 1)


git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5866 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-04-24 11:55:39 +00:00