Marown/yacy_search_server
1
0
Fork 0
mirror of https://github.com/yacy/yacy_search_server.git synced 2024-09-21 00:00:13 +02:00
Code Issues Actions Packages Projects Releases Wiki Activity
4,807 Commits 1 Branch 20 Tags 836 MiB
5ce9a100bb
Commit Graph

3199 Commits

Author SHA1 Message Date
orbiter
080cda97ef added another peer selection rule: 
- select also non-robinson (dht-) peers if their peer tags match with search words
- the peer tag '*' can now act as catch-all rule: shall be selected always

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4963 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-06-30 23:04:32 +00:00
orbiter
d37fd064f9 changed peer selection for search targets: 
- less dht targets are selected
- more other peers are selected: all robinson peers with more than one million urls

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4962 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-06-30 22:42:52 +00:00
orbiter
69aac0d74c modified the diskUsage class regarding the following two aspects: 
1. The usage and dependency of the plasmaSwitchboad was used many times in the past but this was
a bad mistake. The classes should be independent from the switchboard to support a better abstraction. Therefore the object was removed. The parameters from the switchboard are computed outside and then handed over.
2. the class is considered as a tightly connected to hardware resources. Classes which handle data that cannot be replicated because it would need to replicate hadware should not support dynamic object allocation, but should be coded as collection of private static methods. Therefore all class objects had been transformed into static private objects.

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4961 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-06-30 21:47:53 +00:00
danielr
da917cf4b1 undo reduced menu 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4960 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-06-30 07:11:13 +00:00
danielr
0c1dc703e4 - set staticIP at startUp 
- added setting for reduced menu (simpleMenu)


git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4959 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-06-29 18:35:15 +00:00
danielr
f7f9ceb967 diskUsage: replaced blocking sleep with semaphore 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4957 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-06-26 12:05:12 +00:00
lotus
4a53649ee7 fixed dht-urls and ranking distribution log statistics 
* NOTE: please have in mind that there can be whitespaces in pathnames

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4956 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-06-26 07:12:03 +00:00
lotus
8d83185cb4 fixed dht-chunks/protocol log statistics 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4955 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-06-25 08:15:07 +00:00
danielr
63eadfdf84 fixed unlimited FileSizeLimit 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4954 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-06-24 19:11:27 +00:00
lotus
2dc7c00c1c fixed indexing log statistics 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4953 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-06-24 07:01:04 +00:00
danielr
dba7ba079e fixed NPE seen with queues_p.xml (serverClassLoader finds already loaded classes) 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4952 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-06-23 16:55:46 +00:00
det
273fb01142 revert last fix; was wrong 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4950 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-06-21 21:07:28 +00:00
det
b6f50851fa fix memory requirement calculation 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4949 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-06-21 20:58:57 +00:00
lotus
ac85c52bae better readability for MIN_FREE_DISK_SPACE 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4948 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-06-21 10:20:36 +00:00
lotus
54a73b58cf fixed restart on Windows when directory had spaces in it's name 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4947 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-06-21 09:19:26 +00:00
det
609aaf0df3 rework of the windows part 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4943 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-06-20 12:13:06 +00:00
det
1a4f26ba30 exclude HTDOCS from recursiv scan 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4942 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-06-20 10:03:49 +00:00
det
6c07e894d9 add needed sleep 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4941 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-06-20 09:53:23 +00:00
hermens
d742cc080c Fix for RAMCache not flushing 
see: http://forum.yacy-websuche.de/viewtopic.php?f=6&t=1255



git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4940 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-06-19 18:27:48 +00:00
danielr
6b7e873962 resourceObserver refactoring and some synchronisation for console output 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4939 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-06-19 12:40:44 +00:00
orbiter
6bdd99e065 - more asserts to solve the ooB-problem 
- better caching (?), lets see how it behaves

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4937 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-06-18 21:08:56 +00:00
orbiter
b928ae492a some code-cleanup and possible speed enhancements in different core methods 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4935 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-06-17 23:56:39 +00:00
danielr
6a9cc29cdd workaround for IndexOutOfBoundsException in ResultURLs.getExecutorHash() seen @ CrawlResults.html?process=4 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4934 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-06-17 18:56:04 +00:00
orbiter
c998dc6556 - added security functions to flush url and search caches in case that memory is full 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4933 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-06-16 21:39:58 +00:00
orbiter
f4ae8082c3 - better error analysis for ooRange Exception in kelondroBase64Ordering 
- quadcore support for kelondroRowSet array ordering

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4932 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-06-15 23:25:57 +00:00
orbiter
84cbe75005 more asserts 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4930 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-06-15 00:04:59 +00:00
orbiter
e269c12710 small changes in partition routine 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4929 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-06-14 23:17:56 +00:00
orbiter
31efb8fbee - fix for LOG path generation when the DATA/LOG does not exists (fix for bug introduced in SVN 4923) 
- some more/better asserts
- slight performance enhancements in remove method in index management. Works for all who do not run using asserts (the majority)

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4928 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-06-14 22:51:47 +00:00
lotus
877299cc74 better installer on Windows Vista 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4927 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-06-14 18:34:12 +00:00
danielr
726218dd4a fixed logging 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4926 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-06-14 14:10:53 +00:00
orbiter
21c87c36e3 added a log line 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4925 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-06-14 11:57:46 +00:00
danielr
68c38c2d34 - WatchCrawler shows status without JavaScript 
- Performance can be scaled + DHT-profile
- names for pool-threads
- some small refactorings


git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4923 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-06-14 10:24:58 +00:00
lotus
fc79f013c4 better solution to update shortcut 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4920 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-06-12 20:04:32 +00:00
det
c0dfe49743 also exclude collection.0028.commons and RANKING at startup check 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4919 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-06-12 15:17:01 +00:00
det
11656741f1 exclude LOCALE and RELEASE at startup check 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4917 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-06-12 11:25:25 +00:00
lotus
48edbef5c7 * fix: display proper port on 1st startup 
* new message on portchange
* first implementation of external link-update for search page (still inactive)

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4915 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-06-11 19:04:39 +00:00
det
0727bb1e63 rework of console message handling; add of debugging output 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4914 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-06-11 18:43:12 +00:00
lotus
43c47218ef fox for open browser on Windows 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4912 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-06-11 10:18:01 +00:00
orbiter
f5ef7f222e - fixed a bug in parser (directory paths had not been recognized) 
- no access check when a search is made only local without snippet fetch
- added comment and status message in resourceObserver (this takes very long at startup time!)

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4911 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-06-11 09:54:58 +00:00
lotus
ed24eab737 small fix for windows in resource observer 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4909 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-06-10 19:19:54 +00:00
hermens
75b4a5ced4 reinstate old timeout values for transferRWI and transferURL 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4903 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-06-07 23:48:46 +00:00
orbiter
3330181aa0 refactoring: 
find a better way to store BLOBs; generalize current BLOG data structure (kelondroDyn)
and prepare it to replace it with something better. The best candidate is the kelondroHeap,
which will become the kelondroBLOBHeap;
removed also some never-used classes

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4902 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-06-07 23:12:24 +00:00
orbiter
9a9737a54e fix for "no more elements available" exception 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4901 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-06-07 22:07:25 +00:00
lotus
260553c3a5 better messages 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4897 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-06-07 14:24:11 +00:00
det
6afeb535cd another bugfix for the windows drive check 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4896 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-06-07 12:51:07 +00:00
det
b416af7568 bugfix for the windows drive check 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4895 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-06-06 20:38:09 +00:00
danielr
4b71912e76 fixed wrong class name 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4894 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-06-06 17:13:31 +00:00
danielr
7feae906aa - organize imports 
- removed potential null pointer accesses
- removed unnecessary casts


git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4893 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-06-06 16:01:27 +00:00
det
f597185026 Initial import of the resource observer framework 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4892 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-06-06 13:10:21 +00:00
orbiter
c10eaf9bdb - fix for pop-up page upon first start 
- added comments in opensearchdescription to explain fast mode
- release 0.59

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4890 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-06-05 20:46:31 +00:00
danielr
d3037c2950 Accept all SSL-certificates (not only valid and self-signed), but put a warning into log file 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4888 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-06-05 15:21:43 +00:00
orbiter
e0e7f86f82 some bugfixes for the peer-ping process 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4885 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-06-05 12:52:27 +00:00
danielr
cbe722c480 small code cleanUp 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4884 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-06-05 12:41:55 +00:00
orbiter
b21598bdd0 - enhanced handling of own IP address inside seed 
- prevention of false information of own IP address
- enabled searching before an own IP address is assigned (before first ping happened)
- removed warning about limited search function
- added better time-out settings for peer-ping process (10 seconds complete, 5 seconds for back-ping)

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4883 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-06-05 11:01:20 +00:00
orbiter
40d7f485f3 - fixed several NPE bugs 
- fixed loosing of own seed hash (hopefully)
- fixed a bug with crawl start s beginning with (bookmark) files
- added better IP recognition during hello process


git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4882 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-06-04 22:24:00 +00:00
orbiter
2f381b8d7a - fixed at least two causes for a NPE after a use case switch. 
A large refactoring was neccessary
- added another crawl start option: automatic restriction to sub-path
- removed crawlStartSimple and renamed crawl start expert
   to crawl start (without expert)
- some changes to texts in crawl start
- added some more deletions when an web index is deleted:
   delete also queues and robots cache


git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4881 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-06-04 21:34:57 +00:00
orbiter
2a604b7402 added superfast search result computation which can be obtained for local search when snippet fetching is disabled. An example search for the rss interface would be: 
http://localhost:8080/yacysearch.rss?query=yacy&Enter=Search&contentdom=text&count=10&resource=local&verify=false
(just add "&verify=false")

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4878 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-06-03 23:06:01 +00:00
orbiter
9bef20b537 - added cleanup for unused server loggings: they are removed after the client had not been seen since one hour 
- removed configBasic popup trigger when no password is set

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4875 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-06-02 21:49:59 +00:00
orbiter
5f35ba6f05 added a list for content of the brute-force registration hashtable in access tracker 
the purpose of this list is not only to view attempts from read attacks, but also to show if the
yacy-yacy protocol is working with respect to the bug that was fixed with SVN 4869

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4873 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-06-01 21:21:38 +00:00
orbiter
74e3a547db more logging for mySeed loading error cases 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4871 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-06-01 11:24:46 +00:00
orbiter
6f1a3fce05 BF Bugfix 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4869 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-05-29 17:27:38 +00:00
lotus
b09af53643 proper links to directories in repository dirlisting 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4867 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-05-28 19:23:44 +00:00
lotus
ba4091c5b2 proxy sends status code now 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4860 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-05-27 13:11:06 +00:00
orbiter
1a1841392c small fix 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4859 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-05-26 22:55:55 +00:00
orbiter
698293ef32 fix for lonely peers in networks with only one peer, especially intranet peers 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4858 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-05-26 20:48:14 +00:00
orbiter
8a0e401320 - fix for bad code in peer actions 
- fix for bad images in basic config

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4854 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-05-25 22:56:28 +00:00
orbiter
0b52ef3e4b - update of grafics 
- check for startsWith("127.") instead of equals("127.0.0.0")

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4853 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-05-25 20:30:37 +00:00
orbiter
18ad12eceb added another fix for localhost addresses in seeds 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4852 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-05-25 18:41:56 +00:00
orbiter
11e00a0849 - refactoring of seedURL handling 
- additional check for seedURL pointing to localhost: deny such peers

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4851 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-05-25 18:35:38 +00:00
orbiter
e91bf4c8cc - fix for bad reset of index / bad index location after deletion 
- some modification of rssTerminal window location and size

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4850 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-05-24 21:30:22 +00:00
lotus
f284386b63 update deploy improvements for windows - ready for release now :-) 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4849 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-05-24 20:03:11 +00:00
f1ori
fd8bd5d0d1 * fix for http://forum.yacy-websuche.de/viewtopic.php?f=6&t=1176 (encoding issue) 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4848 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-05-24 16:12:16 +00:00
orbiter
25192e0d36 added a deletion button to indexControlRWIs that deletes the complete web index 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4847 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-05-24 12:30:50 +00:00
orbiter
01b3e9431a - fix for http://forum.yacy-websuche.de/viewtopic.php?f=6&t=1140&p=7626#p7626 
- less dots for ppm bar in watchcrawler (one dot for each 10 ppm)

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4846 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-05-24 11:30:16 +00:00
orbiter
8be462986e fix for http://forum.yacy-websuche.de/viewtopic.php?f=6&t=1174&p=7841#p7841 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4845 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-05-24 11:04:44 +00:00
orbiter
f8b015949c fix for bug in html scraper that appears if opening and closing tag are not both in same case 
see http://forum.yacy-websuche.de/viewtopic.php?f=6&t=1173&p=7836#p7836

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4844 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-05-24 10:47:22 +00:00
orbiter
d8277e6af1 - added parsing of numeric html entities for crawler 
- fixed a bug in search response

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4843 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-05-24 10:26:24 +00:00
low012
f06c046f6c *) refactoring (several instances of Indetificator can be created and used parallel now) 
*) fixed a potential NPE


git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4841 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-05-23 18:12:44 +00:00
orbiter
0c173821fd more access security regarding database access and snippet retrieval: restrict number of results for not-authorized searchers 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4838 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-05-23 09:45:33 +00:00
lotus
eac62a6882 * ported restart on Windows to unix-style, works on _noconsole now 
* removed Win9x scripts from build for more tidiness and less decisions for newbies

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4835 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-05-22 15:48:51 +00:00
orbiter
03438ee977 added missing implementation of network-path reference 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4834 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-05-22 00:08:14 +00:00
orbiter
2ba7914f0b fix for NPE exception while fetching remote crawl jobs 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4833 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-05-21 23:07:37 +00:00
lotus
4a48717017 * automatic update for windows 
pleas disable before release because 2nd update fails at the moment
and commandline handling has to be improved for windows
* update via new unTar class
please review stream- and exceptionhandling because I'm fairly new to Java
maybe it can be done concurrent
* updated windows startscripts to values from yacy.init

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4832 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-05-21 15:23:56 +00:00
orbiter
faed00d75d added use cases to basic configuration 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4831 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-05-20 13:21:55 +00:00
orbiter
4229cd275c fixed several details about network switching, default password, random password and localhost authentification 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4830 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-05-20 09:29:01 +00:00
orbiter
d9d1c8de70 more protection against remote shutdown attacks: prevent loading using the crawler 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4829 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-05-19 23:05:19 +00:00
orbiter
c1d721dd2d fix for attacks on localhost-authorized peers from web pages with links to localhost addresses: 
checking of referer in access

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4828 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-05-19 22:17:53 +00:00
orbiter
6aa9bf43f9 fixed server port change 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4826 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-05-19 20:51:09 +00:00
low012
a7dadf7f2f *) first version of a way to determine the language a text is written in (not perfect, but it works) 
*) statistical data of languages can be found in the *.lng files in the new directory called "langfiles"

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4824 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-05-18 21:24:05 +00:00
orbiter
1127d62b64 some enhancements to the access tracker (less synchronization) 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4812 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-05-16 23:33:59 +00:00
orbiter
56a300f92a bugfix / local-search predicate 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4811 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-05-16 21:17:55 +00:00
orbiter
2f29ab8779 more target server access security 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4809 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-05-16 19:50:28 +00:00
orbiter
3bd1db776a implemented switch for admin authorization from localhost: 
- access is granted for localhost users to administration pages by default
- the default setting can be changed in the BasicConfig.html page
- if the BasicConfig page was accessed with post and no password was submitted, a random password is generated
- a headless installation MUST give a password upon first call of the configuration page, otherwise they will not be able to access it again
- if no password is given within 10 minutes after start-up, a random password is generated

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4804 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-05-15 11:26:43 +00:00
orbiter
cfe6790498 - added option to switch between yacy networks, especially between the two default networks (freeworld and intranet), 
from the ConfigNetwork online interface
- to make this possible, a large refactoring and reorganisation of data structures was necessary

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4803 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-05-14 21:36:02 +00:00
orbiter
78087da287 - changed seed file storage to clear text 
- fixed kill script
- fixed saving of seed file (had been corrupted by latest changes)
- some refactoring

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4799 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-05-14 20:30:44 +00:00
orbiter
5fde679acb - fixed problem in performance configuration 
- extended rss fetch size for rssTerminal


git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4798 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-05-13 15:28:55 +00:00
orbiter
db032fb6de - added RWI transmissions to the event terminal 
- fixed bug in Collage
- added 'embedded mode' to collage
- integrated Collage to terminal_p as iframe in embedded mode (Pictures now visible in terminal_p)

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4797 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-05-13 11:46:20 +00:00
orbiter
239cc4428d - better domain graph, faster when more links exist, looks better 
- new authorization rule: localhost is always authorized for administration. This solves many problems with ajax, and also fixed a problem in rssTerminal
- fix bug in RSSFeed which prevented that entries had been recognized as individual, new entries
- added reloading/updating of status image on status page

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4796 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-05-12 22:23:29 +00:00
orbiter
415b92bb07 fix for http://forum.yacy-websuche.de/viewtopic.php?f=6&t=1148&hilit=&p=7711#p7711 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4795 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-05-12 15:06:04 +00:00
apfelmaennchen
2113672bf2 small fix on tag comporator functions 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4794 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-05-12 15:05:27 +00:00
orbiter
dd75b3cabc - patch for bad profiles 
- time-out when deleting profiles

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4793 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-05-12 14:58:56 +00:00
danielr
2080ff72b7 ftpc fix for npe 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4789 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-05-10 22:03:51 +00:00
lotus
e021278bf0 unescape link display in search results 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4788 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-05-10 11:25:54 +00:00
danielr
3f1721b827 informational comment 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4787 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-05-10 11:21:51 +00:00
danielr
04a51b775a changed .org/.net back to America 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4786 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-05-10 11:10:10 +00:00
danielr
74b1a60043 fixed "java.lang.NoClassDefFoundError: org/a" 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4784 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-05-10 08:42:31 +00:00
danielr
f44d5d302b updated TLDs 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4782 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-05-09 12:46:10 +00:00
hermens
5bfc02ccfb Repair publishThread 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4781 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-05-09 11:14:32 +00:00
orbiter
f42c8cf69c updated terminal and dynamic webstructure applet: can now change when crawl is running 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4780 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-05-09 00:01:47 +00:00
orbiter
906c144799 - design update to new terminal and rssTerminal 
- added terminal to main menu
- removed transfer size limitation in server

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4779 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-05-08 21:33:16 +00:00
orbiter
7ec01d444a fix for npe 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4778 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-05-08 20:25:11 +00:00
orbiter
ad0f905124 fix for npe in crawler 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4777 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-05-08 20:16:19 +00:00
danielr
ae03a54d23 pdfParser: updated lib, fixed ClassNotFoundException: CMSError 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4776 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-05-08 16:55:45 +00:00
danielr
0d3808bd9e minor refactoring 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4775 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-05-08 16:51:01 +00:00
orbiter
719f5defb1 updated some grafics at new terminal_p 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4774 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-05-07 23:42:14 +00:00
lotus
9bc56a9edc xss protection 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4772 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-05-07 16:37:13 +00:00
orbiter
b32736762c enhanced rssTerminal 
- 3 lines possible
- distinguishing of private and public data, if not authorized only public data is shown
- shows now more events, including local searches in clear text if user is logged in
- simplyfied peer events
- better recognition of 'real' new peers
- presentation of peer pings from other peers

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4771 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-05-06 23:05:48 +00:00
orbiter
fbb712c669 refactoring: 
moved importer classes to crawler and plasma package

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4770 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-05-06 13:44:38 +00:00
orbiter
1689030ee8 refactoring: moved all crawler classes into their own package 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4768 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-05-06 00:32:41 +00:00
orbiter
d2ba1fd2ab major step forward to network switching (target is easy switch to intranet or other networks .. and back) 
This change is inspired by the need to see a network connected to the index it creates in a indexing team.
It is not possible to divide the network and the index. Therefore all control files for the network was moved to the network within the INDEX/<network-name> subfolder.
The remaining YACYDB is superfluous and can be deleted.
The yacyDB and yacyNews data structures are now part of plasmaWordIndex. Therefore all methods, using static access to yacySeedDB had to be rewritten. A special problem had been all the port forwarding methods which had been tightly mixed with seed construction. It was not possible to move the port forwarding functions to the place, meaning and usage of plasmaWordIndex. Therefore the port forwarding had been deleted (I guess nobody used it and it can be simulated by methods outside of YaCy).
The mySeed.txt is automatically moved to the current network position. A new effect causes that every network will create a different local seed file, which is ok, since the seed identifies the peer only against the network (it is the purpose of the seed hash to give a peer a location within the DHT).
No other functional change has been made. The next steps to enable network switcing are:
- shift of crawler tables from PLASMADB into the network (crawls are also network-specific)
- possibly shift of plasmaWordIndex code into yacy package (index management is network-specific)
- servlet to switch networks 

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4765 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-05-05 23:13:47 +00:00
danielr
d70a472460 added file for previous commit 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4764 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-05-05 05:19:01 +00:00
danielr
d32fe84472 added default User-Agent 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4763 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-05-04 17:26:19 +00:00
danielr
8c5f062e0b corrected YaCy version in HTTP User-Agent 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4762 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-05-04 12:18:00 +00:00
danielr
d7b21bc90c re-added gzip POST for transferRWI/URL (HTTP/1.1 compliant) 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4761 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-05-04 10:53:04 +00:00
low012
8d83febb95 *) BlacklistCleaner_p.java reports exception to log instead of System.err 
*) changes in formatting for better readability in BlacklistCleaner_p.java
*) replaced  test for necessary Java version (was 1.4.2, is 1.5 now)

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4756 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-05-03 10:16:04 +00:00
danielr
d4bce6affd refactoring (initialized static fields, removed empty if/else, serialized some fields in serializable classes) 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4755 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-05-03 09:06:00 +00:00
danielr
be2c9c07ff escape some unescaped characers in URLs (fixes problems with proxy) 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4753 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-05-02 08:19:47 +00:00
orbiter
d0678f7ab9 refactoring as result of 
http://forum.yacy-websuche.de/viewtopic.php?f=6&t=959&p=7560#p7560

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4752 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-05-01 22:40:42 +00:00
orbiter
483e9a2066 - shifted tld recognition methods from yacyURL to serverDomains 
- changed isLocal Property in such a way that it is possible to see if a domain is in the internet (and not intranet)

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4751 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-04-30 23:06:42 +00:00
orbiter
a3df23659c re-implementation of charset checking 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4750 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-04-30 13:23:05 +00:00
orbiter
75a1702133 - fix for ConcurrentModificationException during shutdown 
- fix for Ranking distribution problem (suma-lab peer does not exist any more)

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4749 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-04-30 11:19:52 +00:00
orbiter
32b5b057b9 - modified, simplified old kelondroHTCache object; I believe it should be replaced by something completely new 
- removed tree data type in kelondroHTCache
- added new class kelondroHeap; may be the core for a storage object that will once replace the many-files strategy of kelondroHTCache
- removed compatibility mode in indexRAMRI


git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4747 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-04-29 22:31:05 +00:00
danielr
ec84a52adb change for problem with NPE (seen as "PROXY Unknown Error while processing request") 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4745 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-04-29 16:06:54 +00:00
orbiter
5813cc149f fix for bad rssTerminal behavior 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4744 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-04-28 20:34:37 +00:00
orbiter
88216c1f1f fix for http://forum.yacy-websuche.de/viewtopic.php?f=6&t=1103&hilit=&p=7362#p7362 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4743 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-04-26 22:59:20 +00:00
orbiter
d0b893523e - protection against RAM overflow caused by new peer rss news 
- more XSS protection

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4742 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-04-26 22:53:04 +00:00
orbiter
685794e7e7 fix for parser/encoding Exception 
see http://forum.yacy-websuche.de/viewtopic.php?f=5&t=1111&hilit=&sid=55a320b54e1e3bda9410e7c50b5147f1&p=7431#p7431

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4741 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-04-26 22:14:45 +00:00
orbiter
cf042e6957 reverted change by mistake in yacyVersion 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4740 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-04-26 01:08:59 +00:00
orbiter
9935e83c86 added new news window into the status page. At this moment it is just a test. 
The news inside the window are about peer arrivals and departures, remote search accesses and crawls

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4739 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-04-26 01:00:10 +00:00
orbiter
bac38cfa18 added very rudimentary peer news as rss feed. An example can be retrieved with 
http://localhost:8080/xml/feed.rss?channel=PEERNEWS
to be extended and integrated in interface ...

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4738 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-04-24 23:30:13 +00:00
orbiter
6495227ad6 the class rssReader is replaced by RSSReader, RSSFeed and RSSMessage 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4737 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-04-24 21:45:43 +00:00
orbiter
724bbdf9b2 refactoring of RSS reader 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4736 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-04-24 21:31:07 +00:00
orbiter
b9a2a2d287 more search performance hacks 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4735 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-04-24 15:09:06 +00:00
orbiter
ff755fb858 small corrections and enhancements after search timing profiling 
search should be a little bit faster now

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4734 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-04-24 13:31:55 +00:00
orbiter
e024e3b9cf added new default profiles to distinguish snippet fetch for local and global search 
the difference is, that a local search will no not cause a re-indexing of loaded pages

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4731 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-04-24 08:42:08 +00:00
danielr
9b03310f8a bin jetzt wach :/ 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4729 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-04-23 07:50:21 +00:00
danielr
7bd8601f04 delete old releases compatible with java 1.5 ;) 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4728 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-04-23 07:22:20 +00:00
danielr
da386a1924 fixed deleteOldDownloads if there are no downloads 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4726 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-04-22 21:36:52 +00:00
danielr
21418a22a3 removed DEBUG output 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4725 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-04-22 17:14:34 +00:00
danielr
79a3edeeef deleting downloaded releases after x days (default 30) 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4724 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-04-22 16:53:53 +00:00