Marown/yacy_search_server
1
0
Fork 0
mirror of https://github.com/yacy/yacy_search_server.git synced 2024-09-19 00:01:41 +02:00
Code Issues Actions Packages Projects Releases Wiki Activity
7,295 Commits 1 Branch 20 Tags 836 MiB
9b25d07295
Commit Graph

7295 Commits

This Branch
This Branch
All Branches
Author SHA1 Message Date
orbiter
5892fff51f introduction of dht-burst modes: this can expand the number of target peers in some cases where a better heuristic is needed. The problematic cases are either when a muti-word search is made (still a hard case for our term-oriented DHT) or when a network operator wants that all robinson peers are asked. We therefore introduced two new network steering values that switch on more peers during the peer selection. Because the number of peers can now be very large, the number of maximum httpc connections was also increased. 
Please see new coments in yacy.network.freeworld.unit for details of the new DHT selection methods.
The number of maximum peers is now not fixed to a specific number but may increase with
- the partition exponent
- the number of redundant peers
- the robinson burst percentage
- the multiword burst percentage
The maximum can then be the number of senior peers (all visible peers).

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7479 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2011-02-13 17:37:28 +00:00
orbiter
4588b5a291 - fixed document number limitation for crawls that restrict the number of documents per domain 
- some restructuring of the document counting and logging structures was necessary
- better abstraction of CrawlProfiles
- added deletion of logs to the index deletion option (if the index is deleted using the servlets) which is necessary to reset the domain counters for the page limitation
- more refactoring to get the LibraryProvider more clean
- some refactoring of the Condenser class

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7478 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2011-02-12 00:01:40 +00:00
low012
64f32e8f00 *) replaced all IPs in IP filters for proxy with the proper regular expression 
*) some cleanup

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7477 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2011-02-11 23:37:13 +00:00
lotus
9f6e3f9793 fix: same number format display before and after get/post 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7476 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2011-02-10 09:08:44 +00:00
lotus
4716aa7d3a fix display of statistics link on network.html (caused by menu option) 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7475 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2011-02-09 21:32:45 +00:00
sixcooler
3e8b72be50 update to httpclient-4.1 - sorry forgot some 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7474 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2011-02-08 21:53:20 +00:00
sixcooler
45dcfa3460 update to httpclient-4.1 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7473 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2011-02-08 21:46:24 +00:00
f1ori
d38c739b31 * add ant-target deleteData, to simulate fresh yacy installation 
* useful in IDEs like eclipse

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7472 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2011-02-08 09:12:48 +00:00
low012
74b22dfa24 *) fixed bug which affected blacklist entries which consisted of domain _and_ path parts 
*) minor cleanup

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7471 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2011-02-06 18:07:08 +00:00
low012
4843f04247 *) more input from Moubai 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7470 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2011-02-06 16:00:45 +00:00
low012
b9990e722f *) updated French langugage file with input from Moubai (http://yacy-forum.org/viewtopic.php?p=1646) 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7469 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2011-02-05 11:20:25 +00:00
orbiter
93732d6773 increased number of target peers 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7468 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2011-02-04 13:25:28 +00:00
orbiter
70ca7cec8c fix for http://forum.yacy-websuche.de/viewtopic.php?p=21763#p21763 
and another fix for non-working global search when search options are switched off

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7467 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2011-02-03 10:43:09 +00:00
orbiter
fe93caac5a added flags and administration options to show advanced search and to show search result attributes (for each search result) 
Administration can be done at ConfigPortal.html

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7466 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2011-02-02 15:54:13 +00:00
f1ori
01cb3bbaec * fix patchCharsetEncoding-test (patchCharsetEncoding now returns null on input null) 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7465 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2011-02-02 15:28:41 +00:00
orbiter
431f780f41 patch for bad data in url metadata 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7464 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2011-02-02 01:19:25 +00:00
orbiter
0dfb1ef7d2 added custom targets for media search 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7463 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2011-02-02 00:50:06 +00:00
orbiter
5905f912c5 replaced more double types with float 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7462 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2011-02-02 00:22:00 +00:00
orbiter
0cdfb82963 replaced more appearance of double values by float values 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7461 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2011-02-02 00:06:29 +00:00
orbiter
eb12e15738 moved all Double values to Float values because of 
http://www.exploringbinary.com/java-hangs-when-converting-2-2250738585072012e-308/
YaCy does not really need double-precision floating point computation anywhere, so this should not affect any feature

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7460 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2011-02-01 23:49:11 +00:00
orbiter
fe948abd84 added link on ViewProfile.html to EditProfile_p.html to make it easier to customize the peer 
the link appears only if the user has admin rights

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7459 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2011-01-31 19:55:00 +00:00
f1ori
982aa689ef * fix StringIndexOutOfBoundException in WebStructureGraph 
* add better escaping to saveMap and loadMap

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7458 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2011-01-31 14:25:09 +00:00
low012
77715c2b16 *) less HTML errors, but still several left 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7457 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2011-01-28 17:26:47 +00:00
low012
ae10ed5613 *) added a Set to which filter elements are written before mustmatch-filter is created to avoid huge lists of double elements in mustmatch-filter when starting a crawl from a "Link-List of URL" on CrawlStartSite_p.html 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7456 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2011-01-28 16:24:33 +00:00
orbiter
9a1e0158fa better servlet naming in index administration 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7455 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2011-01-28 11:32:31 +00:00
orbiter
88773e4daa changed the default port from 8080 to 8090 
see also: http://forum.yacy-websuche.de/viewtopic.php?p=21683#p21683

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7454 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2011-01-28 10:54:13 +00:00
orbiter
6c35b68f17 - removed 'peerName' property from the yacy settings file because this information is stored in the yacy seed file 
- the own seed file gets the lead for storage of the peer name
- exchanged default peer name generation method with one that does not use the local ip
- default peer names are now strings starting with '_anon'
- added another switch to suppress forwarding to ConfigBasic if the name was already changed
- replaced all usages of the yacy.conf peerName with access to the local seed
- changes to the peer name are now applied directly and not after the next peer ping


git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7453 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2011-01-28 10:12:17 +00:00
mikeworks
8bc66b9e84 de.lng: Updated German translation and cleaned removed lines - no unused strings log entries 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7452 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2011-01-28 04:46:42 +00:00
orbiter
786166041a - added recording of all accessed and submitted servlets 
- this recording is then used to redirect from the Status.html page to BasicConfig in case that servlet was never submitted
- this acts as an addition to the new default pop-up page 'index.html' which offers an administration link to Status.html. For a first-time user this then redirects directly to the former start page BasicConfig.html

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7451 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2011-01-27 11:17:11 +00:00
orbiter
28f669bf0b - fixed/enhanced move to SD/16:9 images (network, web structure) 
- added logging in peer ping to analyse time-consuming elements which could be cause for disappearing peers

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7450 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2011-01-27 10:16:14 +00:00
orbiter
0376f73fdb extended seed list uploader: do not only upload all active peers but also some more peers that are passive but had been active in the last 24 hours 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7449 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2011-01-26 23:21:33 +00:00
orbiter
3fe03f153d - search page becomes default start page (new users are not forced to do configuration since this is not necessary) 
- adjusted top menu on search page (shows less stuff and now also the network graphics)
- adjusted the network page (looks better in when showing no other navigation on top)


git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7448 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2011-01-26 14:58:28 +00:00
orbiter
1bda9e7ed9 option in network servlet to show the servlet with simple or embed menus 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7447 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2011-01-26 13:59:55 +00:00
orbiter
991b92f4ae enhanced network graphic 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7446 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2011-01-26 13:52:46 +00:00
mikeworks
30df013c2b de.lng: Added German translation to Network Scanner menu 
CrawlStartScanner_p.html: Modified HTML code to validate HTML 4.01 Strict, switched selectAll Javasript, added end tags to input, etc ...

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7445 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2011-01-23 18:36:44 +00:00
orbiter
0887a1c86f better table layout in Network and API Administration 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7444 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2011-01-22 10:53:36 +00:00
orbiter
59d9fe1bd7 added more php mime types 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7443 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2011-01-22 09:52:36 +00:00
orbiter
3ae8f40fc8 removed yacy.network.group - this feature was never used 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7442 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2011-01-22 09:50:36 +00:00
orbiter
efb4ca8fa8 modified auto-delete of search failure-words: 
- words are now not deleted from the search index automatically if index receive is switched off
- a flag in the network definition defines if this feature is switched on at all
- the search filter for not-found word references is switched off for server-side remote searches

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7441 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2011-01-22 09:46:00 +00:00
low012
58346b9a76 *) added lock symbol to "API Action Steering" link in menu since the page is protected 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7440 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2011-01-22 02:11:18 +00:00
lotus
79c5b35a6a fix bug in installer in case there was a 32 bit jre installed on a 64 bit windows. now the necessary 64 bit jre gets installed. 
danke PCA42! http://forum.yacy-websuche.de/viewtopic.php?p=21631#p21631

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7439 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2011-01-21 18:09:24 +00:00
orbiter
f1f03d8c90 more logging for strange network loading bug 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7438 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2011-01-19 09:31:56 +00:00
f1ori
4e29e9712a * create cleanupjob for cached failed urls 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7437 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2011-01-17 15:04:00 +00:00
f1ori
a321c7673d * adminAccountForLocalhost only for localhost 
* yacy crawls local domains also, if no password is set (the interface is already protected)
* it's not required anymore, to set a password in intranet mode

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7436 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2011-01-17 11:37:30 +00:00
lotus
c9cd03b532 adding convenient option for contacting peer on network page 4. can be used on creating new network. this page is hidden from the interface (not linked). 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7435 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2011-01-15 18:49:51 +00:00
hermens
930cb412dd Let SHORT_MILSEC_FORMATTER make a new formatted String every millisecond 
see http://forum.yacy-websuche.de/viewtopic.php?f=6&t=3103


git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7434 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2011-01-15 13:16:40 +00:00
orbiter
090c73e32e catch a OOM in HeapReader iteration 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7433 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2011-01-12 12:04:18 +00:00
low012
48463c4507 *) General private License? ;-) 
*) minor code changes

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7432 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2011-01-12 00:28:08 +00:00
orbiter
c93f4dda72 - cleaned up yacy news 
- removed unused methods
- avoid news generation in case that the peer runs in robinson mode

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7431 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2011-01-12 00:00:14 +00:00
orbiter
6c1b14c8e1 - more control in access tracker: count number of returned search results (not only info how much is in the index) 
- extended query params for this
- enhanced cora

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7430 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2011-01-11 22:58:14 +00:00