Marown/yacy_search_server
1
0
Fork 0
mirror of https://github.com/yacy/yacy_search_server.git synced 2024-09-21 00:00:13 +02:00
Code Issues Actions Packages Projects Releases Wiki Activity
5,724 Commits 1 Branch 20 Tags 836 MiB
28b86385cd
Commit Graph

3664 Commits

Author SHA1 Message Date
orbiter
eacf95213a fix for crawling of mailto-links 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5906 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-05-01 07:25:55 +00:00
orbiter
9c6ac43f66 fixes for wiki parser 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5905 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-04-30 22:03:35 +00:00
orbiter
3a64c9d02f - fix for problem with concurrency when computing word hashes 
- fix for search in case that a urlfilter was used and zero results were returned

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5904 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-04-29 22:14:12 +00:00
orbiter
d3f8aa5a2a set of small fixes 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5903 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-04-29 21:36:20 +00:00
low012
78ffb61297 *) got rid of unnecessary variable which might also fix IndexOutOfBoundsException 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5902 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-04-29 21:08:44 +00:00
orbiter
d31e6f9c14 fix for http://forum.yacy-websuche.de/viewtopic.php?p=14457#p14457 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5899 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-04-29 11:18:17 +00:00
orbiter
8d6212233b fix for IODispatcher 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5896 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-04-28 07:24:28 +00:00
orbiter
f678472f46 fix for quote problem in json output 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5895 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-04-27 22:27:02 +00:00
orbiter
d079d6dfdb small changes in surrogate reader, wiki code and portal test 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5894 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-04-27 20:30:43 +00:00
orbiter
07f09742bb set of small fixes and comments 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5893 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-04-27 15:29:50 +00:00
borg-0300
06ed4ef7b3 * better picture handling 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5891 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-04-27 11:19:15 +00:00
orbiter
5a634cab23 removed generation of anchor link sets in document types that describe container formats. 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5890 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-04-27 08:46:11 +00:00
low012
f1244264b8 *) hopefully fixed bug reported in http://forum.yacy-websuche.de/viewtopic.php?t=2057 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5882 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-04-26 16:18:14 +00:00
orbiter
2e3186189b fix for mediawikiIndex surrogate producer + added concurrency 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5880 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-04-25 21:52:21 +00:00
apfelmaennchen
6f5ea7b1a8 small fix for previous post 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5879 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-04-25 21:28:08 +00:00
apfelmaennchen
138a0747e3 added serverObjects.putJSON as JSON has very particulare encoding requirements 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5877 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-04-25 20:56:29 +00:00
orbiter
d977dd9a96 fix for surrogate loader 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5870 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-04-24 22:54:40 +00:00
orbiter
9cb68353da fix for bug in ProfilingGraph for ppm >> 10000 ppm (!) 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5868 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-04-24 13:18:20 +00:00
orbiter
9e4db75aac reduced internal logging and reduced memory that internal logging can use 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5867 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-04-24 12:09:04 +00:00
orbiter
c10c257255 attempt to fix a deadlock situation where the IODispatcher did not work. 
I suspect the dispatcher thread has crashed and queues filled so no indexing process was able to write data.
This fix tries to heal the problem, but I am unsure if it helps. To get a better view of the problem, some more log outputs had been inserted.
Added also a new attribut indexer.threads to get a control over the number of default threads for the indexer (default is 1)


git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5866 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-04-24 11:55:39 +00:00
orbiter
09987e93fd fixed some more bad handling of byte[] 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5865 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-04-23 22:02:12 +00:00
orbiter
1bcc1450cb more explaining error message in case of IOExceptions during html parsing 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5864 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-04-23 21:18:01 +00:00
orbiter
fe51f4d668 less synchronization may help to prevent deadlocks 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5863 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-04-23 20:54:13 +00:00
orbiter
58802e4201 added missing success test in storeDocumentIndex, 
see http://forum.yacy-websuche.de/viewtopic.php?f=5&t=1922&hilit=

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5862 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-04-23 20:38:56 +00:00
orbiter
171e62bee5 addition to the fix from last commit (which did not work) 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5860 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-04-23 16:36:21 +00:00
orbiter
059949a0d1 tried to fix problem with snippet fetch for second search page when verify=false 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5859 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-04-23 15:29:30 +00:00
lotus
b08991e278 moved some constants, rename of Tray class 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5858 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-04-23 13:18:59 +00:00
orbiter
138422990a - removed useCell option: the indexCell data structure is now the default index structure; old collection data is still migrated 
- added some debugging output to balancer to find a bug
- removed unused classes for index collection handling
- changed some default values for the process handling: more memory needed to prevent OOM

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5856 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-04-22 22:39:12 +00:00
orbiter
1b9e532c87 some concurrency for wikipedia dump reader 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5855 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-04-22 17:43:27 +00:00
lotus
25d2160288 small fix 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5853 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-04-22 13:19:37 +00:00
orbiter
16baa7ad24 To translate a mediawiki dump into the YaCy surrogate format do the following: 
- download a wikipedia dump, i.e. dewiki-20090311-pages-articles.xml.bz2
from http://download.wikimedia.org/dewiki/20090311/
- move dewiki-20090311-pages-articles.xml.bz2 to DATA/HTCACHE/
- start the conversion; open a command shell, move to the yacy home directory and execute
java -Xmx2000m -cp classes:lib/bzip2.jar de.anomic.tools.mediawikiIndex -convert DATA/HTCACHE/dewiki-20090311-pages-articles.xml.bz2 DATA/SURROGATES/in/ http://de.wikipedia.org/wiki/

this generates a series of files to DATA/SURROGATES/in

if YaCy is running (it may run concurrently), it fetches all new dumps in the surrogate-in directory. The export process is transaction-save, that means YaCy will not start reading a dump while the dump is not completely finished.


git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5851 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-04-21 22:12:19 +00:00
orbiter
0b2c98edc9 some more work on the wikipedia-dump exporter (not finished yet) 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5850 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-04-21 15:19:32 +00:00
orbiter
5195c94838 two patches for performance enhancements of the index handover process from documents to the index cache: 
- one word prototype is generated for each document, that is re-used when a specific word is stored.
- the index cache uses now ByteArray objects to reference to the RWI instead of byte[]. This enhances access to the the map that stores the cache. To dump the cache to the FS, the content must be sorted, but sorting takes less time than maintenance of a sorted map during caching.


git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5849 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-04-21 14:23:04 +00:00
orbiter
9416f5c26f more speed test cases: kelondro provides map functions that are more than 20% faster than standard java classes and use less than halve of the memory of java classes: 
just start IndexTest (here with 1000000 test objects)

Performance test: comparing HashMap, TreeMap and kelondroRow
generated 1000000 test data entries 


STANDARD JAVA CLASS MAPS 

sorted map
time   for TreeMap<byte[]> generation: 2110
time   for TreeMap<byte[]> test: 2516, 0 bugs
memory for TreeMap<byte[]>: 29 MB

unsorted map
time   for HashMap<String> generation: 1157
time   for HashMap<String> test: 1516, 0 bugs
memory for HashMap<String>: 61 MB


KELONDRO-ENHANCED MAPS 

sorted map
time   for kelondroMap<byte[]> generation: 1781
time   for kelondroMap<byte[]> test: 2452, 0 bugs
memory for kelondroMap<byte[]>: 15 MB

unsorted map
time   for HashMap<ByteArray> generation: 828
time   for HashMap<ByteArray> test: 953, 0 bugs
memory for HashMap<ByteArray>: 9 MB

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5847 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-04-21 09:29:08 +00:00
orbiter
b53790abb1 more performance hacks: 10% more speed for Base64.compare() which is really often used in YaCy code 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5846 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-04-21 07:39:21 +00:00
orbiter
8ffb9889e1 some fixes and performance hacks 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5845 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-04-20 23:01:44 +00:00
orbiter
dfb96ecb72 more fixes 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5844 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-04-20 22:08:38 +00:00
orbiter
1b8d346b4c fixes in connection with transiton to byte[] hashes 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5843 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-04-20 21:54:00 +00:00
f1ori
0b0a46d35a * fix transferRWI as suggested by celle (thanks!) 
see http://forum.yacy-websuche.de/viewtopic.php?f=5&t=2000#p14023


git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5842 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-04-20 19:51:20 +00:00
orbiter
996572de95 quickfix 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5841 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-04-20 16:11:35 +00:00
orbiter
380ed2dac0 performance and debugging additions 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5840 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-04-20 15:01:43 +00:00
lotus
635b0a9da7 code-split 
allow cgi indexing

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5839 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-04-20 13:28:28 +00:00
orbiter
fa3adbbfc6 added domain checks to surrogate reader and RWI transfer receiver to prevent spaming using surrogates 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5837 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-04-20 06:38:28 +00:00
f1ori
76af84d732 * add custom comparator to ScoreCluster for byte[] 
* fixes http://forum.yacy-websuche.de/viewtopic.php?f=6&t=2010


git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5836 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-04-19 20:01:46 +00:00
lotus
ab0030d7a7 allow dht-out for remote-crawl processing peers on default settings 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5834 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-04-18 20:04:01 +00:00
low012
d1116c049f *) added new method "contains()" to Blacklist interface 
*) implemented contains() in class AbstractBlacklist
*) used new method in Blacklist_p to prevent double entries in blacklists

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5832 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-04-18 16:27:17 +00:00
f1ori
08445e42f0 * don't throw exception, in case of bad charset in http-header 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5831 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-04-18 15:38:29 +00:00
f1ori
2f860a2564 * convert byte[] hashes to string for log output 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5830 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-04-18 14:35:18 +00:00
f1ori
d93a2a6552 * ignore whitespaces so you can copy&paste signatures better 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5828 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-04-17 14:52:42 +00:00
orbiter
fbcbcc5bdb export of yacy document objects as dublin core record in xml 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5826 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-04-17 14:20:12 +00:00
orbiter
d7cbf4cdd4 more performance hacks: less overhead in word hash computation 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5825 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-04-17 13:47:06 +00:00
orbiter
29e96c1a60 bugfixes and performance hacks 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5824 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-04-17 13:04:56 +00:00
orbiter
4e97a31009 corrections in dublin core syntax 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5823 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-04-17 12:23:00 +00:00
f1ori
44daec7936 * introduce signatures to autoupdate 
as long as there aren't publickeys for the updatelocations set,
  no signatures are checked
* wiki-article follows...


git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5822 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-04-17 09:58:06 +00:00
orbiter
538e375901 replaced old caching method for computed word hashes with a better method. The word hash computation is a new performance bottleneck (after the IO bottleneck was removed with the IndexCell data structure) and a better caching for word hashes was necessary. 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5821 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-04-17 09:26:16 +00:00
orbiter
9e853e1977 partly reverting SVN 5818: identical comparator required for join operator 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5820 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-04-17 08:18:01 +00:00
orbiter
e16c25ddf7 (peak-) performance hacks 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5819 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-04-16 22:45:39 +00:00
orbiter
63cd152969 fixes 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5818 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-04-16 22:18:35 +00:00
orbiter
7dfe7e7cc6 fixed some problems with surrogate reader. This is now ready for testing. 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5817 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-04-16 21:29:41 +00:00
orbiter
3a1364ed5c removed example lines from SurrogateReader sources; added additional example file 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5816 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-04-16 21:05:34 +00:00
orbiter
9050a3c4c5 alpha version of surrogate reading and indexing. 
see the example file for an explanation.

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5815 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-04-16 20:47:55 +00:00
orbiter
b15b059c0d fix for latest commit 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5813 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-04-16 19:53:21 +00:00
orbiter
c8624903c6 full redesign of index access data model: 
terms (words) are not any more retrieved by their word hash string, but by a byte[] containing the word hash.
this has strong advantages when RWIs are sorted in the ReferenceContainer Cache and compared with the sun.java TreeMap method, which needed getBytes() and new String() transformations before.
Many thousands of such conversions are now omitted every second, which increases the indexing speed by a factor of two.

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5812 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-04-16 15:29:00 +00:00
f1ori
dd6b5005ff * fix missing charset handling in getpageinfo_p 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5811 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-04-16 12:31:28 +00:00
orbiter
bd5f4c78d8 - added default profile for surrogate indexing 
- integrated surrogate indexing into indexing queue process

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5810 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-04-16 08:01:38 +00:00
orbiter
ad78e3a59f - less lines in rssTerminal 
- crawl more documents: if remote crawling is enabled, a remote crawl list is also loaded if a local crawl is running in case that the indexer is idle

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5809 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-04-15 23:07:51 +00:00
orbiter
bc80dc913a added new surrogate reader (surrogates are parsed documents on batches) 
this will open a new way to insert indexes to YaCy (instead crawling)

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5808 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-04-15 15:30:25 +00:00
orbiter
12d81e98eb - fixed bad search results when searching for empty string 
- simplified result handling and page composition in case that nothing was searched


git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5807 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-04-15 11:22:43 +00:00
orbiter
8a24350036 - fix for join method with new generalized RWI data structure (caused by latest commit) 
- added more functions to mediawiki parser


git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5806 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-04-15 10:26:24 +00:00
orbiter
e58320a507 added more info in log fore debugging 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5805 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-04-15 07:37:36 +00:00
orbiter
89ec3acb3e - full abstraction of index content type: the kelondro full text index may now also contain indexes about other content than text, i.e. navigation indexes or reverse linking indexes. 
- during index joins all word positions are maintained: better ranking for word distance possible; exact phrase match can be implemented soundly


git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5804 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-04-15 06:34:27 +00:00
borg-0300
7a48090fcf - fix for "uk" language 
- svn attributes added

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5803 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-04-14 11:40:44 +00:00
orbiter
dc2af61bc9 allow up to 50 results from remote peers 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5802 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-04-13 21:47:57 +00:00
orbiter
c0e8ed5461 fixed problem with not http client 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5801 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-04-13 21:21:47 +00:00
orbiter
8862a2fed0 ups 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5799 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-04-12 10:22:21 +00:00
orbiter
de68948bc5 better handling of free memory computation and emrgency cache flush for index cell 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5798 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-04-12 09:24:32 +00:00
f1ori
fcb77c3140 * added .im (Isle of Man) to TLD-list 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5794 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-04-10 23:06:48 +00:00
orbiter
b81c7467d8 protection against too many files in RICELL in case of massive emergency dumps caused by low memory 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5791 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-04-09 23:55:47 +00:00
orbiter
d4d87d90c4 - extended experimental wikipedia dump parser 
- removed historic, possibly unused code from wiki parser that was in conflict with actual wikipedia wiki code

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5790 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-04-09 14:55:20 +00:00
orbiter
c3aff2521e fix for NPE 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5789 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-04-09 13:32:56 +00:00
orbiter
57c00dd8c9 fix for bad filtering of common http error 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5788 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-04-09 13:20:09 +00:00
orbiter
14361f1ca4 added log message for index generation in HeapReader 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5787 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-04-09 10:34:22 +00:00
orbiter
c08f9b36a4 refactoring of wiki parser. 
This was done to prepare the wiki parser as parser for wikipedia dumps, which will be used for performance test (to omit crawling)

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5785 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-04-08 15:28:45 +00:00
orbiter
44e01afa5b - refactoring 
- a little bit more abstraction
- new interfaces for index abstraction

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5783 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-04-07 09:34:41 +00:00
orbiter
82fb60a720 increased memory limit for emergency cache flush 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5782 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-04-06 15:54:19 +00:00
low012
9180617dd9 *) Classes to handle import of lists (especially blacklists) from XML files, not used yet, but will be used soon. 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5780 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-04-05 13:36:44 +00:00
lotus
596e6215dc fix in case of white space in path name 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5779 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-04-03 16:07:24 +00:00
orbiter
b887f4a116 keep more free mem 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5778 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-04-03 14:27:04 +00:00
orbiter
c2359f20dd refactoring: better abstraction of reference and metadata prototypes. 
This is a preparation to introduce other index tables as used now only for reverse text indexes. Next application of the reverse index is a citation index.
Moved to version 0.74

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5777 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-04-03 13:23:45 +00:00
orbiter
ab656687d7 more strict BLOB initialization .. may also help to save some ram 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5776 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-04-03 12:42:24 +00:00
orbiter
5b138ada16 fixes to web structure reference collection and url construction 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5775 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-04-03 08:29:40 +00:00
orbiter
a29a11e526 added evaluation of incoming links in webstructure api 
the api hash changed, new XML schema.

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5774 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-04-03 07:59:49 +00:00
orbiter
f6691411b5 - migration of files from SplitTable (which are used for the URL-DB) to a different file name format. 
- the file generation logic is slightly different: files may now have only a maximum size of one gigabyte and a maximum age of one month.

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5773 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-04-02 22:15:33 +00:00
shostakovich
1f37cc6107 Robots.txt is now reused after one day. See forum-topic: 
http://forum.yacy-websuche.de/viewtopic.php?f=5&t=1669&p=13565#p13565

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5772 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-04-02 15:29:36 +00:00
orbiter
f21a8c9e9c a different naming scheme for BLOBArray files. This may be necessary if blobs are written more often than once in a second. 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5771 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-04-02 15:08:56 +00:00
orbiter
7ba078daa1 - added fast site-operator 
- refactoring merge into BLOBArray

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5770 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-04-02 13:26:47 +00:00
orbiter
b4126432bc hardening of index dump write process 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5769 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-04-02 12:24:15 +00:00
orbiter
9bfb2641db - removed deprecated threads 
- added automatic http client reset. this was necessary because excessive intranet crawling caused deadlocks. this hack solved the problem.

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5768 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-04-01 20:13:57 +00:00
orbiter
293290c317 fix for bad assert in last commit 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5767 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-04-01 15:17:14 +00:00
orbiter
bd409fb7ba added web structure analysis for a special domain that can be requested from the api. 
Example:
http://localhost:8080/api/webstructure.xml?about=www.yacy.net
returns a xml with the following content:

<?xml version="1.0"?>
<webstructure>
<domains reference="reverse" count="1" maxref="300">
<domain host="www.yacy.net" id="FXg39Q" date="20090401">
  <citation host="java.sun.com" id="o-R3yY" count="1" />
  <citation host="yacy-suche.de" id="-KCLaB" count="1" />
  <citation host="suma-ev.de" id="VRAHIA" count="1" />
  <citation host="www.kit.edu" id="EMaLDQ" count="1" />
  <citation host="yacy.net" id="Fh1hyQ" count="1" />
  <citation host="www.fzk.de" id="V2Kl-A" count="1" />
  <citation host="en.wikipedia.org" id="rwtdfR" count="3" />
  <citation host="vimeo.com" id="MmdQDY" count="3" />
  <citation host="liebel.fzk.de" id="sX4ozA" count="6" />
</domain>
</domains>
</webstructure>


git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5766 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-04-01 14:53:23 +00:00
orbiter
b6c2167143 - patch for bad web structure dumps 
- added automatic slow down of accessed to specific domains when access to a web page fails

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5765 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-04-01 13:21:47 +00:00
orbiter
0139988c04 - added writing of temporary file names and renaming to final file name when index dump/merge are done. Interrupted merges can be cleaned up. 
- added clean-up of unfinished merges and unused idx/gap files
- enhanced merge file selection method

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5764 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-04-01 12:39:11 +00:00
orbiter
3621aa96ab - added a memory protection for the IndexCell migration 
- fix for bad cell file selection

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5763 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-03-31 19:17:45 +00:00
orbiter
568e8f1741 fix in unmountBLOB 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5762 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-03-31 17:03:13 +00:00
orbiter
9da69d6b68 - better selection of files to be merged 
- fix for getChannel().close(), which works on windows but not on macs and linux

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5761 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-03-31 16:49:02 +00:00
orbiter
d39a5b42ca more care about open file handles. Now files also close on windows and can be deleted afterwards. 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5760 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-03-31 12:42:12 +00:00
orbiter
029495e64d fixed bug introduced in SVN 5756 in EcoTable.put() 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5759 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-03-31 07:51:32 +00:00
orbiter
587838bd09 git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5758 6c8d7289-2bf4-0310-a012-ef5d649a1542 2009-03-30 21:13:53 +00:00
orbiter
d2e2420a68 - added another file selection method for index cell merge 
- more hacks to check that files are closed propertly and filehandles do not exist after files are closed.

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5757 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-03-30 19:05:08 +00:00
orbiter
96eaecda3e - added migration class to go from index collections to the index cell data structure. 
- added better control over file deletion, because this sometimes fails, especially on windows

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5756 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-03-30 15:31:25 +00:00
orbiter
0f0b4aec75 better index cell merge logic 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5754 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-03-30 06:22:27 +00:00
orbiter
832fef670f migration of urls-files into subdirectory METADATA 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5753 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-03-30 04:41:06 +00:00
orbiter
fa07234d4e fix for clear method: now deletes files 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5752 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-03-29 21:28:14 +00:00
lulabad
df87e4dbf6 missing count of send Index and URLs 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5747 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-03-28 20:49:58 +00:00
borg-0300
c450e3746b svn attributes added 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5736 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-03-20 15:44:59 +00:00
orbiter
37f892b988 added new concurrent merger class for IndexCell RWI data 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5735 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-03-20 14:54:37 +00:00
borg-0300
8c494afcfe svn attributes added 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5734 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-03-20 11:21:32 +00:00
orbiter
67aaffc0a2 - added Latency control to the crawler: 
because of the strongly enhanced indexing speed when using the new IndexCell RWI data structures (> 2000PPM on my notebook), it is now necessary to control the crawling speed depending on the response time of the target server (which is also YaCy in case of some intranet indexing use cases).
The latency factor in crawl delay times is derived from the time that a target hosts takes to answer on http requests. For internet domains, the crawl delay is a minimum of twice the response time, in intranet cases the delay time is now a halve of the response time.

- added API to monitor the latency times of the crawler:
a new api at /api/latency_p.xml returns the current response times of domains, the time when the domain was accessed by the crawler the last time and many more attributes.

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5733 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-03-20 10:21:23 +00:00
orbiter
0926310461 another performance hack 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5731 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-03-18 22:33:36 +00:00
orbiter
ebe5d69d14 performance hacks 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5730 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-03-18 22:19:08 +00:00
orbiter
61f9dbf0cc - fixed a display problem in watch crawler 
- another small enhancement in balancer

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5729 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-03-18 21:25:52 +00:00
orbiter
b3f75e48fa - enhanced balancer: auto-solving of waiting-deadlocks 
- removed deprecated cache-init size value
- more debug lines for IndexCell cache dump merge

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5728 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-03-18 20:21:19 +00:00
orbiter
9a90ea05e0 added a merge operation for IndexCell data structures 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5727 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-03-18 16:14:31 +00:00
orbiter
d99ff745aa fix for http://forum.yacy-websuche.de/viewtopic.php?p=13378#p13378 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5726 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-03-18 10:29:13 +00:00
orbiter
0c3ab291c4 fix for http://forum.yacy-websuche.de/viewtopic.php?p=13354#p13354 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5725 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-03-17 22:20:58 +00:00
orbiter
a9cea419ef Integration of the new index data structure IndexCell 
This is the start of a testing phase for IndexCell data structure which will replace
the collections and caching strategy. IndexCall creation and maintenance is fast, has
no caching overhead, very low IO load and is the basis for the next data structure,
index segments.

IndexCell files are stored at DATA/<network>/TEXT/RICELL
With this commit still the old data structures are used, until a flag in yacy.conf is set.
To switch to the new data structure, set
useCell = true
in yacy.conf. Then you will have no access any more to TEXT/RICACHE and TEXT/RICOLLECTION

This code is still bleeding-edge development. Please do not use the new data structure for
production now. Future versions may have changed data types, or other storage locations.
The next main release will have a migration feature for old data structures.

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5724 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-03-17 13:03:27 +00:00
borg-0300
fd0976c0a7 refactoring 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5723 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-03-16 18:08:43 +00:00
orbiter
83792d9233 more refactoring 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5722 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-03-16 16:24:53 +00:00
borg-0300
ce79239322 "typo" 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5721 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-03-16 16:22:33 +00:00
borg-0300
cdbdc731c5 small updates: unescape, isCGI 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5720 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-03-16 08:49:49 +00:00
orbiter
474aac65af more refactoring 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5719 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-03-16 08:32:28 +00:00
orbiter
209f25f5f5 refactoring to integrate indexCell data structures 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5718 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-03-16 00:18:37 +00:00
borg-0300
359a238acf faster isCGI() 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5717 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-03-14 13:47:49 +00:00
borg-0300
f75628e53b some corrections 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5716 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-03-14 11:08:32 +00:00
orbiter
b7138e5fcb even more efficient comparator calls (less System.arraycopy for primary keys) 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5715 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-03-14 00:41:20 +00:00
orbiter
65784eb656 - more efficient comparator calls 
- fix for http://forum.yacy-websuche.de/viewtopic.php?p=13331#p13331

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5714 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-03-14 00:07:37 +00:00
orbiter
44874cb550 added a deleteOnExit for blob file deletion in case that a deletion is not successful. 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5713 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-03-13 22:47:31 +00:00
orbiter
66f78d67e0 bad idea. Concurrency in index management will be done differently 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5712 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-03-13 22:22:11 +00:00
orbiter
7dff1cba62 removed option to use different primary keys in kelondro tables 
this option was never used and there is also no use to set other columns but the first as the primary key. as a result, access methods to the key do not need to compute key positions, and they work faster.

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5711 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-03-13 16:52:31 +00:00
orbiter
7f67238f8b refactoring of plasmaWordIndex: less methods in the class, separated the index to CachedIndexCollection 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5710 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-03-13 14:56:25 +00:00
orbiter
14a1c33823 refactoring of wordIndex class 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5709 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-03-13 10:34:51 +00:00
orbiter
d49238a637 more performance hacks: better default values for scaling, less memory usage 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5708 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-03-13 10:07:04 +00:00
orbiter
39644dc14e performance hacks to compare methods in database core 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5707 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-03-13 09:30:19 +00:00
orbiter
e2e7949feb replaced old PPM computation with a better one that simply sums up events that had been stored in the profiling table. 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5706 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-03-13 00:13:47 +00:00
orbiter
f6d989aa04 added new class RowSetArray which arranges RowSet objects like Elements in a hashtable, but still provides the functionality of sorted enumeration. The new class is now integrated into the ObjectIndexCache, which is the core class to provide index functions to all database files. The new index access is about twice as fast as before. This has strong speed enhancement effects on all parts of YaCy. 
The speed of the kelondro indexing class ObjectIndexCache can be compared with Javas standard TreeMap with the main method in IntegerHandleIndex. The result is, that the kelondro indexing needs only 1/5 of the memory that TreeMap uses! In exchange, the kelondro classes are slower than TreeMap, about four (!) times slower. However, this is not so bad because the better use of the memory is a strong advantage and makes it possible that YaCy can maintain such a large number of document (> 50 million) in one peer.

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5705 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-03-12 23:05:18 +00:00
borg-0300
0a2fabeef3 static TMPDIR 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5704 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-03-12 16:23:12 +00:00
lotus
9f7e62e900 refactoring 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5703 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-03-12 16:20:04 +00:00
lotus
f35dc11dc4 allow crawl start from pages with script tags 
http://forum.yacy-websuche.de/viewtopic.php?f=5&t=1910

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5702 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-03-12 16:12:50 +00:00
orbiter
6958eff196 removed unnecessary exceptions, extended testing in IntegerHandleIndex 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5701 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-03-12 07:35:17 +00:00
orbiter
13c666adef performance hack to ObjectIndex put() method: 
Java standard classes provide a Map Interface, that has a put() method that returns the object that was replaced by the object that was the argument of the put call. The kelondro ObjectIndex defined a put method in the same way, that means it also returned the previous value of the Entry object before the put call. However, this value was not used by the calling code in the most cases. Omitting a return of the previous value would cause some performance benefit. This change implements a put method that does not return the previous value to reflect the common use. Omitting the return of previous values will cause some benefit in performance. The functionality to get the previous value is still maintained, and provided with a new 'replace' method. 

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5700 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-03-11 20:23:19 +00:00