Marown/yacy_search_server
1
0
Fork 0
mirror of https://github.com/yacy/yacy_search_server.git synced 2024-09-21 00:00:13 +02:00
Code Issues Actions Packages Projects Releases Wiki Activity
4,653 Commits 1 Branch 20 Tags 836 MiB
474659a71f
Commit Graph

2988 Commits

Author SHA1 Message Date
orbiter
474659a71f - modified and enhanced the crawl balancer: better list export, fixing of damaged crawl queue at start-up, re-sorting at start-up to enhance domain order 
- added option to set minimum crawl delta for domains in balancer
- added default values to crawl deltas in yacy.init
- added configuration for these deltas in performance queues
- enhanced performance setting computation (more time for indexing queue for a faster flush
- remote crawling is now enabled during local crawling if indexer has space and time for more links
- added database stub for new distributed file system
- refactoring of time computation to get an abstraction level that will be used by a TTL rule in new distributed file system

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4966 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-07-03 13:08:37 +00:00
orbiter
080cda97ef added another peer selection rule: 
- select also non-robinson (dht-) peers if their peer tags match with search words
- the peer tag '*' can now act as catch-all rule: shall be selected always

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4963 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-06-30 23:04:32 +00:00
orbiter
d37fd064f9 changed peer selection for search targets: 
- less dht targets are selected
- more other peers are selected: all robinson peers with more than one million urls

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4962 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-06-30 22:42:52 +00:00
orbiter
69aac0d74c modified the diskUsage class regarding the following two aspects: 
1. The usage and dependency of the plasmaSwitchboad was used many times in the past but this was
a bad mistake. The classes should be independent from the switchboard to support a better abstraction. Therefore the object was removed. The parameters from the switchboard are computed outside and then handed over.
2. the class is considered as a tightly connected to hardware resources. Classes which handle data that cannot be replicated because it would need to replicate hadware should not support dynamic object allocation, but should be coded as collection of private static methods. Therefore all class objects had been transformed into static private objects.

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4961 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-06-30 21:47:53 +00:00
danielr
da917cf4b1 undo reduced menu 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4960 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-06-30 07:11:13 +00:00
danielr
0c1dc703e4 - set staticIP at startUp 
- added setting for reduced menu (simpleMenu)


git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4959 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-06-29 18:35:15 +00:00
danielr
f7f9ceb967 diskUsage: replaced blocking sleep with semaphore 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4957 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-06-26 12:05:12 +00:00
lotus
4a53649ee7 fixed dht-urls and ranking distribution log statistics 
* NOTE: please have in mind that there can be whitespaces in pathnames

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4956 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-06-26 07:12:03 +00:00
lotus
8d83185cb4 fixed dht-chunks/protocol log statistics 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4955 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-06-25 08:15:07 +00:00
danielr
63eadfdf84 fixed unlimited FileSizeLimit 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4954 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-06-24 19:11:27 +00:00
lotus
2dc7c00c1c fixed indexing log statistics 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4953 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-06-24 07:01:04 +00:00
danielr
dba7ba079e fixed NPE seen with queues_p.xml (serverClassLoader finds already loaded classes) 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4952 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-06-23 16:55:46 +00:00
det
273fb01142 revert last fix; was wrong 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4950 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-06-21 21:07:28 +00:00
det
b6f50851fa fix memory requirement calculation 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4949 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-06-21 20:58:57 +00:00
lotus
ac85c52bae better readability for MIN_FREE_DISK_SPACE 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4948 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-06-21 10:20:36 +00:00
lotus
54a73b58cf fixed restart on Windows when directory had spaces in it's name 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4947 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-06-21 09:19:26 +00:00
det
609aaf0df3 rework of the windows part 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4943 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-06-20 12:13:06 +00:00
det
1a4f26ba30 exclude HTDOCS from recursiv scan 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4942 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-06-20 10:03:49 +00:00
det
6c07e894d9 add needed sleep 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4941 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-06-20 09:53:23 +00:00
hermens
d742cc080c Fix for RAMCache not flushing 
see: http://forum.yacy-websuche.de/viewtopic.php?f=6&t=1255



git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4940 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-06-19 18:27:48 +00:00
danielr
6b7e873962 resourceObserver refactoring and some synchronisation for console output 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4939 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-06-19 12:40:44 +00:00
orbiter
6bdd99e065 - more asserts to solve the ooB-problem 
- better caching (?), lets see how it behaves

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4937 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-06-18 21:08:56 +00:00
orbiter
b928ae492a some code-cleanup and possible speed enhancements in different core methods 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4935 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-06-17 23:56:39 +00:00
danielr
6a9cc29cdd workaround for IndexOutOfBoundsException in ResultURLs.getExecutorHash() seen @ CrawlResults.html?process=4 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4934 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-06-17 18:56:04 +00:00
orbiter
c998dc6556 - added security functions to flush url and search caches in case that memory is full 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4933 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-06-16 21:39:58 +00:00
orbiter
f4ae8082c3 - better error analysis for ooRange Exception in kelondroBase64Ordering 
- quadcore support for kelondroRowSet array ordering

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4932 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-06-15 23:25:57 +00:00
orbiter
84cbe75005 more asserts 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4930 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-06-15 00:04:59 +00:00
orbiter
e269c12710 small changes in partition routine 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4929 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-06-14 23:17:56 +00:00
orbiter
31efb8fbee - fix for LOG path generation when the DATA/LOG does not exists (fix for bug introduced in SVN 4923) 
- some more/better asserts
- slight performance enhancements in remove method in index management. Works for all who do not run using asserts (the majority)

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4928 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-06-14 22:51:47 +00:00
lotus
877299cc74 better installer on Windows Vista 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4927 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-06-14 18:34:12 +00:00
orbiter
21c87c36e3 added a log line 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4925 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-06-14 11:57:46 +00:00
danielr
68c38c2d34 - WatchCrawler shows status without JavaScript 
- Performance can be scaled + DHT-profile
- names for pool-threads
- some small refactorings


git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4923 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-06-14 10:24:58 +00:00
lotus
fc79f013c4 better solution to update shortcut 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4920 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-06-12 20:04:32 +00:00
det
c0dfe49743 also exclude collection.0028.commons and RANKING at startup check 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4919 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-06-12 15:17:01 +00:00
det
11656741f1 exclude LOCALE and RELEASE at startup check 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4917 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-06-12 11:25:25 +00:00
lotus
48edbef5c7 * fix: display proper port on 1st startup 
* new message on portchange
* first implementation of external link-update for search page (still inactive)

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4915 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-06-11 19:04:39 +00:00
det
0727bb1e63 rework of console message handling; add of debugging output 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4914 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-06-11 18:43:12 +00:00
lotus
43c47218ef fox for open browser on Windows 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4912 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-06-11 10:18:01 +00:00
orbiter
f5ef7f222e - fixed a bug in parser (directory paths had not been recognized) 
- no access check when a search is made only local without snippet fetch
- added comment and status message in resourceObserver (this takes very long at startup time!)

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4911 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-06-11 09:54:58 +00:00
lotus
ed24eab737 small fix for windows in resource observer 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4909 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-06-10 19:19:54 +00:00
hermens
75b4a5ced4 reinstate old timeout values for transferRWI and transferURL 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4903 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-06-07 23:48:46 +00:00
orbiter
3330181aa0 refactoring: 
find a better way to store BLOBs; generalize current BLOG data structure (kelondroDyn)
and prepare it to replace it with something better. The best candidate is the kelondroHeap,
which will become the kelondroBLOBHeap;
removed also some never-used classes

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4902 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-06-07 23:12:24 +00:00
orbiter
9a9737a54e fix for "no more elements available" exception 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4901 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-06-07 22:07:25 +00:00
lotus
260553c3a5 better messages 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4897 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-06-07 14:24:11 +00:00
det
6afeb535cd another bugfix for the windows drive check 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4896 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-06-07 12:51:07 +00:00
det
b416af7568 bugfix for the windows drive check 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4895 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-06-06 20:38:09 +00:00
danielr
4b71912e76 fixed wrong class name 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4894 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-06-06 17:13:31 +00:00
danielr
7feae906aa - organize imports 
- removed potential null pointer accesses
- removed unnecessary casts


git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4893 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-06-06 16:01:27 +00:00
det
f597185026 Initial import of the resource observer framework 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4892 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-06-06 13:10:21 +00:00
danielr
d3037c2950 Accept all SSL-certificates (not only valid and self-signed), but put a warning into log file 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4888 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-06-05 15:21:43 +00:00