Marown/yacy_search_server
1
0
Fork 0
mirror of https://github.com/yacy/yacy_search_server.git synced 2024-09-21 00:00:13 +02:00
Code Issues Actions Packages Projects Releases Wiki Activity
7,231 Commits 1 Branch 20 Tags 836 MiB
619b561a4a
Commit Graph

4455 Commits

Author SHA1 Message Date
orbiter
619b561a4a enhanced secondary search: index abstracts decompression is now much faster and does not cause strong CPU load after several searches with more than one word 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7565 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2011-03-07 23:12:39 +00:00
orbiter
27ecdb5444 use less peers for remote search 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7561 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2011-03-07 21:24:46 +00:00
orbiter
cb1f49d0f2 replaced all 'new String' with default encoding (missing) or UTF-8 encoding with a String generation method that uses a pre-defined Charset constant for UTF-8. This avoids a cache-lookup for the Charset object using String hashing of the String 'UTF-8'. 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7558 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2011-03-07 20:36:40 +00:00
orbiter
7138f4036b less synchronization, better thread dump tool 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7556 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2011-03-07 15:29:45 +00:00
orbiter
8d14916c74 more patches for a better out-of-memory management 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7555 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2011-03-07 01:45:11 +00:00
orbiter
c2c5b12882 - even less memory for circle tool 
- background thread for bookmark initialization: this uses a DNS lookup which may cause long waiting times during startup

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7554 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2011-03-06 12:30:22 +00:00
orbiter
799c534935 one more patch again OOM during secondary remote search 
see also: http://forum.yacy-websuche.de/viewtopic.php?f=6&t=3202

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7551 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2011-03-05 19:52:34 +00:00
orbiter
77b1e921a9 this asserts prevents a network operation in case of sabotage and must be removed therefore 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7550 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2011-03-04 14:34:37 +00:00
orbiter
bed79402be introduction of a new remote search load control: the remote search has taken 10 results per peer with a time-out of 3 seconds so far. The attributes of number of results per peer and time-out time can now be configured. 
This has two aspects: the user who searches may want to increase these values to get more results and more load on the remote side and the user of the server which is accessed for this search may want to restrict the load. Both sides can now be configured. The server-site maximum load parameters are defined by a network definition and the client-side search request load can be defined by each user individually but when the remote search is done the requested service is limited to the network definition.

You can find now in the network definition file:
network.unit.remotesearch.maxcount and network.unit.remotesearch.maxtime
and in the yacy.conf file:
remotesearch.maxcount and remotesearch.maxtime

There is currently no web interface to define the client-side remote search attributes, please set them manually
    

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7548 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2011-03-04 13:44:00 +00:00
orbiter
6dfaf6fef7 fix for bug in deletion of old seeds 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7547 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2011-03-04 10:00:37 +00:00
orbiter
993b9bc1a8 memory/performance hacks, less synchronization, better concurrency 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7544 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2011-03-03 11:30:04 +00:00
orbiter
42d90664f3 - fixed a memory leak in the httpc.post method (no finish) 
- patched some more memory-saving relevant code
- some more minor bug fixes

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7541 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2011-03-01 09:03:33 +00:00
orbiter
38dce547c0 better concurrency (less locking on date formatting) more logging and minor bug fixes 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7540 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2011-02-28 06:28:29 +00:00
f1ori
59dea3a284 * implement url proxy, a proxy via the url http://peer:port/proxy.html?url=http://domain.tld/path 
* enable with proxyURL = true
* could be useful to browse specific pages with proxy or use own improvements in proxy

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7538 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2011-02-27 21:39:38 +00:00
mikeworks
8b7b783c49 Tray.java: Broke the build on with wrong non UTF-8 encoded file and french umlauts (unmappable character for encoding UTF8) 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7537 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2011-02-27 15:01:46 +00:00
mikeworks
db65ada467 Tray.java: Added localization for french tray icon command - although this can probably also done better than with if statements. (preferably also from the locales file) 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7536 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2011-02-27 11:42:33 +00:00
orbiter
89d337841c more logging for OOMs 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7534 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2011-02-26 09:27:15 +00:00
orbiter
b1781d7aae some more performance hacks 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7533 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2011-02-26 01:24:49 +00:00
orbiter
5e186e0122 continuing the fight against deadlocks during time formatting: better caching. 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7531 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2011-02-25 21:11:53 +00:00
orbiter
dec24244cf added convenience class to generate UTF StringBody objects with a default UTF8 charset. 
Reason: if this is not used in StringBody-Class initialization, a default charset name is parsed.
This is a synchronized process and all classes using default charsets synchronize at that point
Synchronization is omitted if this class is used
 

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7530 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2011-02-25 13:26:09 +00:00
orbiter
1110d16af9 performance hack: replaced generic row.getColBytes() call with row.getPrimaryKeyBytes() where the column is 0 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7529 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2011-02-25 12:41:27 +00:00
orbiter
19b2a50578 - enhanced date formatter cache 
- added more instances of formatter objects to different classes to make them independent in case of lockings that may applay during synchronization of the date formatter object (date formatting is not thread-safe and must be synchronized therefore)

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7528 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2011-02-25 12:23:00 +00:00
orbiter
f2e8ffd768 enhancement in synchronisation 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7525 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2011-02-25 01:19:39 +00:00
orbiter
ad7fcb9d61 Enhanced Base64Order transformation: less overhead (transformation between StringBuilder and byte[]) 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7523 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2011-02-25 00:56:31 +00:00
orbiter
0ce17d823a - fixed bug in ordering 
- fixed ConcurrentModificationException in set join


git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7519 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2011-02-24 10:32:46 +00:00
orbiter
dec4f36700 - fix for missing favicons in search widgets 
- fix for bad digest/hash computation in case of interrupts to class

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7518 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2011-02-23 23:04:05 +00:00
orbiter
e3ef4e3021 - increased default peer ping time from 2 minutes to 1 minute 
- filtering out too old peers when reading seed lists (limit is now 240 minutes)
- added concurrent host names resolving in front of the http client because the http client uses the java built-in DNS resolve which is not multithreading-safe (i have seen deadlocks in thread dumps showing that this bug in jdk is still there)

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7515 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2011-02-23 09:42:01 +00:00
orbiter
cd19d0517e added dns resolve to HTTPClient POST using a dns cache to prevent that that not-thread-safe built-in dns cache inside apache http client is used 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7513 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2011-02-22 22:58:19 +00:00
orbiter
d28f8040e0 removed unnecessary recording function that caused also a performance problem after serving too much files 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7512 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2011-02-22 13:33:28 +00:00
orbiter
af87af0d4c - removed synchronization in serverSwitch which should improve speed 
- fixed wrong assert in network graph
- enhanced double check method in table class

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7511 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2011-02-22 12:56:25 +00:00
orbiter
4bd65532da initialization of libraries concurrently (faster start-up) 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7510 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2011-02-22 10:57:38 +00:00
orbiter
57e6728cb7 - removed usage of /etc/alternatives/www-browser because of problems with lynx, see: 
http://forum.yacy-websuche.de/viewtopic.php?p=21959#p21959
  please look if the browser that is linked with /etc/alternatives/www-browser can be detected and insert call again if
  it can be made sure that this does not call lynx
- replaced severe warnings with just warnings in yacyClient

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7506 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2011-02-21 11:53:45 +00:00
orbiter
d84b4a072e healing for some OOM problems 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7502 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2011-02-21 00:38:49 +00:00
orbiter
82f262f685 - enhanced circle drawing speed 
- beautified 'moving dot' feature (using smaller and correctly positioned dots)
- added moving dots to DHT transfer

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7500 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2011-02-21 00:03:11 +00:00
orbiter
29dc416ac6 more animations in graphics. See network and access picture. 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7498 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2011-02-19 01:56:12 +00:00
orbiter
a80ee9a03d THE GRID is coming to YaCy .. see new animated graphics on http://localhost:8090/AccessGrid_p.html 
showing incoming and outgoing connections in an animated way

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7496 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2011-02-18 23:19:35 +00:00
low012
ce012e11aa *) deleted LogStatistics since the page did not work anymore and it seemed to be obsolete, tell me if you miss it and I will add it again 
*) a few minor changes

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7494 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2011-02-18 01:46:07 +00:00
low012
c5051c4020 *) fixed bug which caused entries to not be deleted when deleting by URL on IndexCreateWWWLocalQueue_p.html (I hope this did not break anything else) 
*)  cleaned up code a little bit

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7493 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2011-02-18 01:25:46 +00:00
orbiter
d58071947a maybe terminateOldSessions is too slow, removed sleep 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7492 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2011-02-17 23:36:46 +00:00
orbiter
6c52e31993 new methods to open a browser 
- if YaCy is started with the option -gui, it is not in headless mode. Then the java 1.6 browse method is used if all other methods fail
- in linux, the path /etc/alternatives/www-browser is used if no firefox is installed

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7480 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2011-02-14 16:15:14 +00:00
orbiter
5892fff51f introduction of dht-burst modes: this can expand the number of target peers in some cases where a better heuristic is needed. The problematic cases are either when a muti-word search is made (still a hard case for our term-oriented DHT) or when a network operator wants that all robinson peers are asked. We therefore introduced two new network steering values that switch on more peers during the peer selection. Because the number of peers can now be very large, the number of maximum httpc connections was also increased. 
Please see new coments in yacy.network.freeworld.unit for details of the new DHT selection methods.
The number of maximum peers is now not fixed to a specific number but may increase with
- the partition exponent
- the number of redundant peers
- the robinson burst percentage
- the multiword burst percentage
The maximum can then be the number of senior peers (all visible peers).

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7479 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2011-02-13 17:37:28 +00:00
orbiter
4588b5a291 - fixed document number limitation for crawls that restrict the number of documents per domain 
- some restructuring of the document counting and logging structures was necessary
- better abstraction of CrawlProfiles
- added deletion of logs to the index deletion option (if the index is deleted using the servlets) which is necessary to reset the domain counters for the page limitation
- more refactoring to get the LibraryProvider more clean
- some refactoring of the Condenser class

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7478 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2011-02-12 00:01:40 +00:00
low012
64f32e8f00 *) replaced all IPs in IP filters for proxy with the proper regular expression 
*) some cleanup

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7477 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2011-02-11 23:37:13 +00:00
orbiter
93732d6773 increased number of target peers 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7468 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2011-02-04 13:25:28 +00:00
orbiter
70ca7cec8c fix for http://forum.yacy-websuche.de/viewtopic.php?p=21763#p21763 
and another fix for non-working global search when search options are switched off

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7467 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2011-02-03 10:43:09 +00:00
orbiter
fe93caac5a added flags and administration options to show advanced search and to show search result attributes (for each search result) 
Administration can be done at ConfigPortal.html

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7466 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2011-02-02 15:54:13 +00:00
orbiter
5905f912c5 replaced more double types with float 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7462 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2011-02-02 00:22:00 +00:00
orbiter
0cdfb82963 replaced more appearance of double values by float values 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7461 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2011-02-02 00:06:29 +00:00
orbiter
eb12e15738 moved all Double values to Float values because of 
http://www.exploringbinary.com/java-hangs-when-converting-2-2250738585072012e-308/
YaCy does not really need double-precision floating point computation anywhere, so this should not affect any feature

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7460 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2011-02-01 23:49:11 +00:00
f1ori
982aa689ef * fix StringIndexOutOfBoundException in WebStructureGraph 
* add better escaping to saveMap and loadMap

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7458 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2011-01-31 14:25:09 +00:00