Marown/yacy_search_server
1
0
Fork 0
mirror of https://github.com/yacy/yacy_search_server.git synced 2024-09-19 00:01:41 +02:00
Code Issues Actions Packages Projects Releases Wiki Activity
8,376 Commits 1 Branch 20 Tags 836 MiB
461a0ce052
Commit Graph

528 Commits

Author SHA1 Message Date
Michael Peter Christen
461a0ce052 removed warnings 2012-06-05 20:03:43 +02:00
Michael Peter Christen
407fdf6968 more bug fixes and performance hacks for search process 2012-06-05 15:04:23 +02:00
Michael Peter Christen
a1fe65b115 performance hacks 2012-06-05 12:06:26 +02:00
Michael Peter Christen
e0d8643226 - performance hacks 
- added log warnings in case that search processes run into time-out
situations
- better concurrency for Integer formatter (used a non-synchronized
formatter before)
- bugfix for search termination (a poison pill was missing)
- added timeout parameters for search (again) -> target is, that they
are never reached.
 2012-06-04 15:37:39 +02:00
Michael Peter Christen
9b4c699526 ehanced location search: 
- search request are now made using a map boundary
- search results are only computed for the map boundary
- the number of results is adopted to the results in the visible range
- added a double-buffering for the search result markers
- added a search query option for the search results:
/radius/<lat>/<lon>/<radius>
 2012-05-31 22:39:53 +02:00
Michael Peter Christen
1f48d1528b performance hacks 2012-05-31 00:46:30 +02:00
Michael Peter Christen
10da7335ea performance hack: use a hash cache for all hashes that are computed by a 
byte array. If this hash is used in a HashMap (which is very often the
case) then this hack eliminates a lot of re-computations of the same
hash.
 2012-05-30 16:59:13 +02:00
Michael Peter Christen
7c1feefb28 introduced a default 10 second time-out in rwi normalization time 
uring search process to prevent endless deadlocks after a very long
running search
 2012-05-30 16:26:05 +02:00
Michael Peter Christen
8d997d55b6 better logging 2012-05-30 15:47:35 +02:00
Michael Peter Christen
43c2c6e588 better logging 2012-05-30 15:27:45 +02:00
Michael Peter Christen
c15fcde1c8 add-on to latest commit 2012-05-21 17:52:30 +02:00
Michael Peter Christen
cf47d94888 performance hack to parse numbers inside of substrings without actually 
generating a substring. This avoids the allocation of a String object
ech time a substring is parsed. Should affect CPU load during RWI
transmission.
 2012-05-21 13:40:46 +02:00
Michael Peter Christen
7e0ddbd275 added a "fromCache" flag in Response object to omit one cache.has() 
check during snippet generation. This should cause less blockings
 2012-05-21 03:03:47 +02:00
Michael Peter Christen
c6a09eab0b synchronization needed 2012-05-21 00:58:29 +02:00
reger
6696cb1313 bugfix: lookup of peernames no result for active peer in page IndexControlRWIs_p.html -> Transfer RWI to other Peer 
SeedDB.lookupByName searche for lowercase peerNames, while MapColumnIndex.getIndex uses peername as is in the keyset.
Changed the index init to insert lowercase peer names as key
 2012-05-20 05:25:16 +02:00
Michael Peter Christen
f294f2e295 bugfix to http://bugs.yacy.net/view.php?id=181 
tried to make a bit less 'noise' to dns server

also included: less processes in snippet fetch to reduce load during
search on small computers
 2012-05-19 01:06:33 +02:00
Michael Peter Christen
acf8d521a2 fix for http://bugs.yacy.net/view.php?id=126 2012-05-19 00:21:03 +02:00
Michael Peter Christen
fa735f4f04 Merge branch 'master' of ssh://git@gitorious.org/yacy/rc1.git 2012-05-17 23:40:08 +02:00
Michael Peter Christen
3e1bc9477f Merge branch 'master' of ssh://git@gitorious.org/yacy/rc1.git 2012-05-17 13:58:09 +02:00
Michael Peter Christen
6f8a2fef1f small speed enhancement using a column factory 2012-05-17 11:08:48 +02:00
Roland 'Quix0r' Haeder
d10627d591 More sync in close() methods 
Conflicts:
	source/net/yacy/kelondro/logging/GuiHandler.java
	source/net/yacy/kelondro/workflow/InstantBusyThread.java
 2012-05-17 06:03:18 +02:00
Roland 'Quix0r' Haeder
fbb946f913 Made a method static (Eclipse suggested it), removed unused import, pk=null check does now output a warning in logfile 2012-05-17 05:55:44 +02:00
Michael Peter Christen
89142d1e8d removed (not all) warnings 2012-05-16 13:42:32 +02:00
Michael Peter Christen
15db703808 added missing serialization to remove all warnings 2012-05-15 13:13:07 +02:00
Michael Peter Christen
1795a7325b made HandleSet serializable 2012-05-15 12:55:15 +02:00
Roland 'Quix0r' Haeder
a093ccf5eb Now used synchronization in all close() methods to make sure all objects 
are 'closed' in an ordered way

Conflicts:
	source/de/anomic/http/server/ChunkedInputStream.java
	source/de/anomic/http/server/ChunkedOutputStream.java
	source/de/anomic/http/server/ContentLengthInputStream.java
	source/net/yacy/cora/protocol/Domains.java
	source/net/yacy/cora/services/federated/solr/SolrShardingConnector.java
	source/net/yacy/cora/services/federated/solr/SolrSingleConnector.java
	source/net/yacy/document/content/dao/PhpBB3Dao.java
	source/net/yacy/document/parser/html/AbstractTransformer.java
	source/net/yacy/kelondro/blob/BEncodedHeap.java
	source/net/yacy/kelondro/blob/HeapReader.java
	source/net/yacy/kelondro/index/RAMIndexCluster.java
	source/net/yacy/kelondro/io/ByteCountInputStream.java
	source/net/yacy/kelondro/logging/ConsoleOutErrHandler.java
	source/net/yacy/kelondro/table/SQLTable.java
 2012-05-14 07:41:55 +02:00
Michael Peter Christen
0cf3d36eae more tolerance in case of corrupted file 2012-05-11 20:46:50 +02:00
Michael Peter Christen
34f4225d7e less 'wellformed' calls without asserts 2012-05-08 23:24:39 +02:00
Michael Peter Christen
ba6aaabc51 refactoring + parser bugfixes 2012-05-04 17:28:27 +02:00
Michael Christen
e32055aa15 added stub classes for 
- a new database for url reference data ('seen links')
- a new database extending the references to the full url metadata
attributes set which shall replace the old metadata database if it is
finished
- migration help classes stub to use old and new metadata databases
simultanously
 2012-04-13 07:09:15 +02:00
Michael Peter Christen
2fc8ecee36 ConcurrentLinkedQueue has a VERY long return time on the .size() method. 
See
http://docs.oracle.com/javase/6/docs/api/java/util/concurrent/ConcurrentLinkedQueue.html

and the following test programm:

public class QueueLengthTimeTest {


    public static long countTest(Queue<Integer> q, int c) {
        long t = System.currentTimeMillis();
        for (int i = 0; i < c; i++) {
            q.add(q.size());
        }
        return System.currentTimeMillis() - t;
    }

    public static void main(String[] args) {
        int c = 1;
        for (int i = 0; i < 100; i++) {
            Runtime.getRuntime().gc();
            long t1 = countTest(new ArrayBlockingQueue<Integer>(c), c);
            Runtime.getRuntime().gc();
            long t2 = countTest(new LinkedBlockingQueue<Integer>(), c);
            Runtime.getRuntime().gc();
            long t3 = countTest(new ConcurrentLinkedQueue<Integer>(),
c);

            System.out.println("count = " + c + ": ArrayBlockingQueue =
" + t1 + ", LinkedBlockingQueue = " + t2 + ", ConcurrentLinkedQueue = "
+ t3);
            c = c * 2;
        }
    }
}
 2012-02-27 00:42:32 +01:00
Michael Peter Christen
213c8d97f2 use less proccesses in process pool 2012-02-25 14:07:20 +01:00
Michael Peter Christen
c639248c23 protection against strange answers from remote peers during search 2012-02-25 14:07:02 +01:00
Michael Peter Christen
1cd711d005 added classes for citation references (for new citation ranking) 2012-02-24 01:07:15 +01:00
Michael Peter Christen
e0f1e7d904 added new citation reference data structure that shall be used for a 
citation ranking
 2012-02-23 01:22:29 +01:00
Michael Peter Christen
e18a4f6b74 more tolerant merge iterator 2012-02-23 01:21:24 +01:00
Michael Peter Christen
7e4e3fe5b6 free some memory after parsing html 2012-02-02 09:55:27 +01:00
Michael Peter Christen
4540174fe0 memory hacks 2012-02-02 07:37:00 +01:00
Michael Peter Christen
b4409cc803 small redesign of blob column index and usage 2012-02-02 06:43:57 +01:00
Michael Peter Christen
d5c1f2746e performance hack 2012-02-02 06:43:15 +01:00
Michael Peter Christen
803963aebd performance hack: better space grow in CharBuffer (speeds up html 
parser)
 2012-02-01 23:27:59 +01:00
Michael Peter Christen
e2f8f263e8 changed storage of search words: keep order 2012-02-01 18:13:31 +01:00
Michael Peter Christen
0b67a0a5d8 added a column index for tables in blob files. This is heavily used 
during receiving of DHT submissions and when answering remote search
requests. Both events together may have caused IO-deadlocking and this
commit shall fix that.
 2012-02-01 15:11:21 +01:00
Michael Peter Christen
e3bb73c3d6 serialized some database access methods 2012-01-31 21:13:49 +01:00
Michael Peter Christen
2ea585d616 fix for host navigator 2012-01-26 18:10:34 +01:00
Michael Peter Christen
ef78f22ee1 performance hack 2012-01-25 12:48:48 +01:00
Michael Peter Christen
a02fdf8625 better error messages 2012-01-23 00:47:25 +01:00
Michael Peter Christen
c6ba44468e timeout = 5000 instead 3000 2012-01-23 00:45:32 +01:00
low012
8776b84c10 *) small fix to make password change function of reconfigureYACY.sh work 
again
 2012-01-17 20:43:19 +01:00
Michael Peter Christen
4901cee3cc suppress auto-tagged subject entries when sending out or receiving 
metadata from other peers
 2012-01-17 02:10:05 +01:00