Marown/yacy_search_server
1
0
Fork 0
mirror of https://github.com/yacy/yacy_search_server.git synced 2024-09-21 00:00:13 +02:00
Code Issues Actions Packages Projects Releases Wiki Activity
4,745 Commits 1 Branch 20 Tags 836 MiB
be28af50f5
Commit Graph

3157 Commits

Author SHA1 Message Date
orbiter
9a9737a54e fix for "no more elements available" exception 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4901 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-06-07 22:07:25 +00:00
lotus
260553c3a5 better messages 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4897 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-06-07 14:24:11 +00:00
det
6afeb535cd another bugfix for the windows drive check 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4896 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-06-07 12:51:07 +00:00
det
b416af7568 bugfix for the windows drive check 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4895 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-06-06 20:38:09 +00:00
danielr
4b71912e76 fixed wrong class name 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4894 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-06-06 17:13:31 +00:00
danielr
7feae906aa - organize imports 
- removed potential null pointer accesses
- removed unnecessary casts


git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4893 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-06-06 16:01:27 +00:00
det
f597185026 Initial import of the resource observer framework 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4892 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-06-06 13:10:21 +00:00
orbiter
c10eaf9bdb - fix for pop-up page upon first start 
- added comments in opensearchdescription to explain fast mode
- release 0.59

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4890 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-06-05 20:46:31 +00:00
danielr
d3037c2950 Accept all SSL-certificates (not only valid and self-signed), but put a warning into log file 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4888 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-06-05 15:21:43 +00:00
orbiter
e0e7f86f82 some bugfixes for the peer-ping process 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4885 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-06-05 12:52:27 +00:00
danielr
cbe722c480 small code cleanUp 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4884 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-06-05 12:41:55 +00:00
orbiter
b21598bdd0 - enhanced handling of own IP address inside seed 
- prevention of false information of own IP address
- enabled searching before an own IP address is assigned (before first ping happened)
- removed warning about limited search function
- added better time-out settings for peer-ping process (10 seconds complete, 5 seconds for back-ping)

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4883 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-06-05 11:01:20 +00:00
orbiter
40d7f485f3 - fixed several NPE bugs 
- fixed loosing of own seed hash (hopefully)
- fixed a bug with crawl start s beginning with (bookmark) files
- added better IP recognition during hello process


git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4882 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-06-04 22:24:00 +00:00
orbiter
2f381b8d7a - fixed at least two causes for a NPE after a use case switch. 
A large refactoring was neccessary
- added another crawl start option: automatic restriction to sub-path
- removed crawlStartSimple and renamed crawl start expert
   to crawl start (without expert)
- some changes to texts in crawl start
- added some more deletions when an web index is deleted:
   delete also queues and robots cache


git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4881 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-06-04 21:34:57 +00:00
orbiter
2a604b7402 added superfast search result computation which can be obtained for local search when snippet fetching is disabled. An example search for the rss interface would be: 
http://localhost:8080/yacysearch.rss?query=yacy&Enter=Search&contentdom=text&count=10&resource=local&verify=false
(just add "&verify=false")

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4878 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-06-03 23:06:01 +00:00
orbiter
9bef20b537 - added cleanup for unused server loggings: they are removed after the client had not been seen since one hour 
- removed configBasic popup trigger when no password is set

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4875 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-06-02 21:49:59 +00:00
orbiter
5f35ba6f05 added a list for content of the brute-force registration hashtable in access tracker 
the purpose of this list is not only to view attempts from read attacks, but also to show if the
yacy-yacy protocol is working with respect to the bug that was fixed with SVN 4869

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4873 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-06-01 21:21:38 +00:00
orbiter
74e3a547db more logging for mySeed loading error cases 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4871 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-06-01 11:24:46 +00:00
orbiter
6f1a3fce05 BF Bugfix 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4869 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-05-29 17:27:38 +00:00
lotus
b09af53643 proper links to directories in repository dirlisting 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4867 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-05-28 19:23:44 +00:00
lotus
ba4091c5b2 proxy sends status code now 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4860 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-05-27 13:11:06 +00:00
orbiter
1a1841392c small fix 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4859 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-05-26 22:55:55 +00:00
orbiter
698293ef32 fix for lonely peers in networks with only one peer, especially intranet peers 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4858 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-05-26 20:48:14 +00:00
orbiter
8a0e401320 - fix for bad code in peer actions 
- fix for bad images in basic config

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4854 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-05-25 22:56:28 +00:00
orbiter
0b52ef3e4b - update of grafics 
- check for startsWith("127.") instead of equals("127.0.0.0")

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4853 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-05-25 20:30:37 +00:00
orbiter
18ad12eceb added another fix for localhost addresses in seeds 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4852 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-05-25 18:41:56 +00:00
orbiter
11e00a0849 - refactoring of seedURL handling 
- additional check for seedURL pointing to localhost: deny such peers

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4851 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-05-25 18:35:38 +00:00
orbiter
e91bf4c8cc - fix for bad reset of index / bad index location after deletion 
- some modification of rssTerminal window location and size

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4850 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-05-24 21:30:22 +00:00
lotus
f284386b63 update deploy improvements for windows - ready for release now :-) 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4849 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-05-24 20:03:11 +00:00
f1ori
fd8bd5d0d1 * fix for http://forum.yacy-websuche.de/viewtopic.php?f=6&t=1176 (encoding issue) 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4848 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-05-24 16:12:16 +00:00
orbiter
25192e0d36 added a deletion button to indexControlRWIs that deletes the complete web index 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4847 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-05-24 12:30:50 +00:00
orbiter
01b3e9431a - fix for http://forum.yacy-websuche.de/viewtopic.php?f=6&t=1140&p=7626#p7626 
- less dots for ppm bar in watchcrawler (one dot for each 10 ppm)

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4846 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-05-24 11:30:16 +00:00
orbiter
8be462986e fix for http://forum.yacy-websuche.de/viewtopic.php?f=6&t=1174&p=7841#p7841 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4845 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-05-24 11:04:44 +00:00
orbiter
f8b015949c fix for bug in html scraper that appears if opening and closing tag are not both in same case 
see http://forum.yacy-websuche.de/viewtopic.php?f=6&t=1173&p=7836#p7836

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4844 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-05-24 10:47:22 +00:00
orbiter
d8277e6af1 - added parsing of numeric html entities for crawler 
- fixed a bug in search response

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4843 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-05-24 10:26:24 +00:00
low012
f06c046f6c *) refactoring (several instances of Indetificator can be created and used parallel now) 
*) fixed a potential NPE


git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4841 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-05-23 18:12:44 +00:00
orbiter
0c173821fd more access security regarding database access and snippet retrieval: restrict number of results for not-authorized searchers 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4838 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-05-23 09:45:33 +00:00
lotus
eac62a6882 * ported restart on Windows to unix-style, works on _noconsole now 
* removed Win9x scripts from build for more tidiness and less decisions for newbies

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4835 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-05-22 15:48:51 +00:00
orbiter
03438ee977 added missing implementation of network-path reference 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4834 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-05-22 00:08:14 +00:00
orbiter
2ba7914f0b fix for NPE exception while fetching remote crawl jobs 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4833 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-05-21 23:07:37 +00:00
lotus
4a48717017 * automatic update for windows 
pleas disable before release because 2nd update fails at the moment
and commandline handling has to be improved for windows
* update via new unTar class
please review stream- and exceptionhandling because I'm fairly new to Java
maybe it can be done concurrent
* updated windows startscripts to values from yacy.init

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4832 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-05-21 15:23:56 +00:00
orbiter
faed00d75d added use cases to basic configuration 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4831 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-05-20 13:21:55 +00:00
orbiter
4229cd275c fixed several details about network switching, default password, random password and localhost authentification 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4830 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-05-20 09:29:01 +00:00
orbiter
d9d1c8de70 more protection against remote shutdown attacks: prevent loading using the crawler 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4829 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-05-19 23:05:19 +00:00
orbiter
c1d721dd2d fix for attacks on localhost-authorized peers from web pages with links to localhost addresses: 
checking of referer in access

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4828 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-05-19 22:17:53 +00:00
orbiter
6aa9bf43f9 fixed server port change 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4826 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-05-19 20:51:09 +00:00
low012
a7dadf7f2f *) first version of a way to determine the language a text is written in (not perfect, but it works) 
*) statistical data of languages can be found in the *.lng files in the new directory called "langfiles"

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4824 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-05-18 21:24:05 +00:00
orbiter
1127d62b64 some enhancements to the access tracker (less synchronization) 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4812 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-05-16 23:33:59 +00:00
orbiter
56a300f92a bugfix / local-search predicate 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4811 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-05-16 21:17:55 +00:00
orbiter
2f29ab8779 more target server access security 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4809 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-05-16 19:50:28 +00:00
orbiter
3bd1db776a implemented switch for admin authorization from localhost: 
- access is granted for localhost users to administration pages by default
- the default setting can be changed in the BasicConfig.html page
- if the BasicConfig page was accessed with post and no password was submitted, a random password is generated
- a headless installation MUST give a password upon first call of the configuration page, otherwise they will not be able to access it again
- if no password is given within 10 minutes after start-up, a random password is generated

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4804 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-05-15 11:26:43 +00:00
orbiter
cfe6790498 - added option to switch between yacy networks, especially between the two default networks (freeworld and intranet), 
from the ConfigNetwork online interface
- to make this possible, a large refactoring and reorganisation of data structures was necessary

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4803 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-05-14 21:36:02 +00:00
orbiter
78087da287 - changed seed file storage to clear text 
- fixed kill script
- fixed saving of seed file (had been corrupted by latest changes)
- some refactoring

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4799 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-05-14 20:30:44 +00:00
orbiter
5fde679acb - fixed problem in performance configuration 
- extended rss fetch size for rssTerminal


git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4798 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-05-13 15:28:55 +00:00
orbiter
db032fb6de - added RWI transmissions to the event terminal 
- fixed bug in Collage
- added 'embedded mode' to collage
- integrated Collage to terminal_p as iframe in embedded mode (Pictures now visible in terminal_p)

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4797 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-05-13 11:46:20 +00:00
orbiter
239cc4428d - better domain graph, faster when more links exist, looks better 
- new authorization rule: localhost is always authorized for administration. This solves many problems with ajax, and also fixed a problem in rssTerminal
- fix bug in RSSFeed which prevented that entries had been recognized as individual, new entries
- added reloading/updating of status image on status page

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4796 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-05-12 22:23:29 +00:00
orbiter
415b92bb07 fix for http://forum.yacy-websuche.de/viewtopic.php?f=6&t=1148&hilit=&p=7711#p7711 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4795 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-05-12 15:06:04 +00:00
apfelmaennchen
2113672bf2 small fix on tag comporator functions 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4794 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-05-12 15:05:27 +00:00
orbiter
dd75b3cabc - patch for bad profiles 
- time-out when deleting profiles

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4793 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-05-12 14:58:56 +00:00
danielr
2080ff72b7 ftpc fix for npe 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4789 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-05-10 22:03:51 +00:00
lotus
e021278bf0 unescape link display in search results 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4788 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-05-10 11:25:54 +00:00
danielr
3f1721b827 informational comment 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4787 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-05-10 11:21:51 +00:00
danielr
04a51b775a changed .org/.net back to America 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4786 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-05-10 11:10:10 +00:00
danielr
74b1a60043 fixed "java.lang.NoClassDefFoundError: org/a" 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4784 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-05-10 08:42:31 +00:00
danielr
f44d5d302b updated TLDs 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4782 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-05-09 12:46:10 +00:00
hermens
5bfc02ccfb Repair publishThread 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4781 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-05-09 11:14:32 +00:00
orbiter
f42c8cf69c updated terminal and dynamic webstructure applet: can now change when crawl is running 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4780 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-05-09 00:01:47 +00:00
orbiter
906c144799 - design update to new terminal and rssTerminal 
- added terminal to main menu
- removed transfer size limitation in server

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4779 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-05-08 21:33:16 +00:00
orbiter
7ec01d444a fix for npe 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4778 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-05-08 20:25:11 +00:00
orbiter
ad0f905124 fix for npe in crawler 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4777 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-05-08 20:16:19 +00:00
danielr
ae03a54d23 pdfParser: updated lib, fixed ClassNotFoundException: CMSError 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4776 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-05-08 16:55:45 +00:00
danielr
0d3808bd9e minor refactoring 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4775 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-05-08 16:51:01 +00:00
orbiter
719f5defb1 updated some grafics at new terminal_p 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4774 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-05-07 23:42:14 +00:00
lotus
9bc56a9edc xss protection 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4772 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-05-07 16:37:13 +00:00
orbiter
b32736762c enhanced rssTerminal 
- 3 lines possible
- distinguishing of private and public data, if not authorized only public data is shown
- shows now more events, including local searches in clear text if user is logged in
- simplyfied peer events
- better recognition of 'real' new peers
- presentation of peer pings from other peers

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4771 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-05-06 23:05:48 +00:00
orbiter
fbb712c669 refactoring: 
moved importer classes to crawler and plasma package

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4770 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-05-06 13:44:38 +00:00
orbiter
1689030ee8 refactoring: moved all crawler classes into their own package 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4768 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-05-06 00:32:41 +00:00
orbiter
d2ba1fd2ab major step forward to network switching (target is easy switch to intranet or other networks .. and back) 
This change is inspired by the need to see a network connected to the index it creates in a indexing team.
It is not possible to divide the network and the index. Therefore all control files for the network was moved to the network within the INDEX/<network-name> subfolder.
The remaining YACYDB is superfluous and can be deleted.
The yacyDB and yacyNews data structures are now part of plasmaWordIndex. Therefore all methods, using static access to yacySeedDB had to be rewritten. A special problem had been all the port forwarding methods which had been tightly mixed with seed construction. It was not possible to move the port forwarding functions to the place, meaning and usage of plasmaWordIndex. Therefore the port forwarding had been deleted (I guess nobody used it and it can be simulated by methods outside of YaCy).
The mySeed.txt is automatically moved to the current network position. A new effect causes that every network will create a different local seed file, which is ok, since the seed identifies the peer only against the network (it is the purpose of the seed hash to give a peer a location within the DHT).
No other functional change has been made. The next steps to enable network switcing are:
- shift of crawler tables from PLASMADB into the network (crawls are also network-specific)
- possibly shift of plasmaWordIndex code into yacy package (index management is network-specific)
- servlet to switch networks 

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4765 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-05-05 23:13:47 +00:00
danielr
d70a472460 added file for previous commit 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4764 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-05-05 05:19:01 +00:00
danielr
d32fe84472 added default User-Agent 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4763 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-05-04 17:26:19 +00:00
danielr
8c5f062e0b corrected YaCy version in HTTP User-Agent 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4762 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-05-04 12:18:00 +00:00
danielr
d7b21bc90c re-added gzip POST for transferRWI/URL (HTTP/1.1 compliant) 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4761 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-05-04 10:53:04 +00:00
low012
8d83febb95 *) BlacklistCleaner_p.java reports exception to log instead of System.err 
*) changes in formatting for better readability in BlacklistCleaner_p.java
*) replaced  test for necessary Java version (was 1.4.2, is 1.5 now)

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4756 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-05-03 10:16:04 +00:00
danielr
d4bce6affd refactoring (initialized static fields, removed empty if/else, serialized some fields in serializable classes) 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4755 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-05-03 09:06:00 +00:00
danielr
be2c9c07ff escape some unescaped characers in URLs (fixes problems with proxy) 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4753 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-05-02 08:19:47 +00:00
orbiter
d0678f7ab9 refactoring as result of 
http://forum.yacy-websuche.de/viewtopic.php?f=6&t=959&p=7560#p7560

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4752 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-05-01 22:40:42 +00:00
orbiter
483e9a2066 - shifted tld recognition methods from yacyURL to serverDomains 
- changed isLocal Property in such a way that it is possible to see if a domain is in the internet (and not intranet)

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4751 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-04-30 23:06:42 +00:00
orbiter
a3df23659c re-implementation of charset checking 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4750 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-04-30 13:23:05 +00:00
orbiter
75a1702133 - fix for ConcurrentModificationException during shutdown 
- fix for Ranking distribution problem (suma-lab peer does not exist any more)

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4749 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-04-30 11:19:52 +00:00
orbiter
32b5b057b9 - modified, simplified old kelondroHTCache object; I believe it should be replaced by something completely new 
- removed tree data type in kelondroHTCache
- added new class kelondroHeap; may be the core for a storage object that will once replace the many-files strategy of kelondroHTCache
- removed compatibility mode in indexRAMRI


git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4747 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-04-29 22:31:05 +00:00
danielr
ec84a52adb change for problem with NPE (seen as "PROXY Unknown Error while processing request") 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4745 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-04-29 16:06:54 +00:00
orbiter
5813cc149f fix for bad rssTerminal behavior 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4744 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-04-28 20:34:37 +00:00
orbiter
88216c1f1f fix for http://forum.yacy-websuche.de/viewtopic.php?f=6&t=1103&hilit=&p=7362#p7362 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4743 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-04-26 22:59:20 +00:00
orbiter
d0b893523e - protection against RAM overflow caused by new peer rss news 
- more XSS protection

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4742 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-04-26 22:53:04 +00:00
orbiter
685794e7e7 fix for parser/encoding Exception 
see http://forum.yacy-websuche.de/viewtopic.php?f=5&t=1111&hilit=&sid=55a320b54e1e3bda9410e7c50b5147f1&p=7431#p7431

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4741 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-04-26 22:14:45 +00:00
orbiter
cf042e6957 reverted change by mistake in yacyVersion 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4740 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-04-26 01:08:59 +00:00
orbiter
9935e83c86 added new news window into the status page. At this moment it is just a test. 
The news inside the window are about peer arrivals and departures, remote search accesses and crawls

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4739 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-04-26 01:00:10 +00:00
orbiter
bac38cfa18 added very rudimentary peer news as rss feed. An example can be retrieved with 
http://localhost:8080/xml/feed.rss?channel=PEERNEWS
to be extended and integrated in interface ...

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4738 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-04-24 23:30:13 +00:00
orbiter
6495227ad6 the class rssReader is replaced by RSSReader, RSSFeed and RSSMessage 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4737 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-04-24 21:45:43 +00:00
orbiter
724bbdf9b2 refactoring of RSS reader 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4736 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-04-24 21:31:07 +00:00
orbiter
b9a2a2d287 more search performance hacks 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4735 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-04-24 15:09:06 +00:00
orbiter
ff755fb858 small corrections and enhancements after search timing profiling 
search should be a little bit faster now

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4734 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-04-24 13:31:55 +00:00
orbiter
e024e3b9cf added new default profiles to distinguish snippet fetch for local and global search 
the difference is, that a local search will no not cause a re-indexing of loaded pages

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4731 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-04-24 08:42:08 +00:00
danielr
9b03310f8a bin jetzt wach :/ 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4729 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-04-23 07:50:21 +00:00
danielr
7bd8601f04 delete old releases compatible with java 1.5 ;) 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4728 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-04-23 07:22:20 +00:00
danielr
da386a1924 fixed deleteOldDownloads if there are no downloads 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4726 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-04-22 21:36:52 +00:00
danielr
21418a22a3 removed DEBUG output 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4725 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-04-22 17:14:34 +00:00
danielr
79a3edeeef deleting downloaded releases after x days (default 30) 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4724 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-04-22 16:53:53 +00:00
danielr
763f9d4f5d serverCore: setting timeout for new connection before SSLDetect 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4723 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-04-22 09:03:16 +00:00
orbiter
1995faef8d - refactoring of Colage back-end: move to plasma package 
- renamed also the plasmaCrawlResults to have a consistent naming for url and image queues
- added a double-check for the images
- added additional queues for the images: all worse-quality images go there, so the queue can be used also if no sizes are given; no image is lost
- added a cleanup for the stacks so they cannot flood the memory

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4722 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-04-21 22:42:49 +00:00
orbiter
d7e89c2aca fixed near-deadlock situation when deleting crawl profiles 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4721 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-04-20 22:10:26 +00:00
orbiter
5e3ce46339 - better logging when rejecting a url because it is not in declared domain 
- more XSS attack protection

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4720 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-04-20 21:36:25 +00:00
danielr
48ffd61e6a changed "patched wrong" to warning, so it goes to the logfile 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4716 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-04-19 07:54:44 +00:00
orbiter
2f629d20a7 - tried to fix the '4217666-problem' 
- removed more unused code

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4715 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-04-19 04:24:29 +00:00
orbiter
512f48e7d6 - removed unused methods 
- fixed xss attack on peer list in CrawlStartSimple

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4714 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-04-19 03:33:07 +00:00
orbiter
3c76342619 - added servlet to configure the search page greeting line 
- added information output about the current network definition in the network servlet
- better description and usage of profile entries in User Profile servlet regarding FOAF format
- reformatting of menues at status page

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4710 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-04-18 13:58:56 +00:00
danielr
d1ee231866 HTTPC close more unused connections 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4702 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-04-15 16:37:51 +00:00
danielr
181796cffb - HTTPC ConnectionInfo entfernen bei Exceptions, unnötigen Code entfernt 
- FTPC (GET-)connections bei Fehlern auf jeden Fall schliessen


git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4701 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-04-15 15:27:32 +00:00
orbiter
04c1226c80 added/fixed missing integrity-test else-case during deploy in case that we update with a tar file 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4700 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-04-15 15:20:35 +00:00
orbiter
45ae3da7e7 another patch to prevent NPE in EcoTable 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4698 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-04-14 05:33:32 +00:00
orbiter
cb93ded5c6 applied configuration path patch 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4697 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-04-14 04:10:51 +00:00
danielr
96e39b297a reduced StackTraces (by connect timed out) 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4696 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-04-14 03:50:49 +00:00
orbiter
93376acdca fixed a bad chunkcache limit check which could have caused ArrayIndexOutOfBoundsExceptions 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4695 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-04-14 03:49:02 +00:00
orbiter
1cab240198 patch for possible NPE in EcoTable iterator 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4694 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-04-14 03:20:37 +00:00
orbiter
9a32a4c328 fixed concurrentModificationException during hello-process 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4693 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-04-14 03:04:28 +00:00
danielr
64c33e717f catched ConcurrentModificationException in ConnectionInfo.cleanUp so cleanUp is not interrupted 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4692 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-04-14 03:02:44 +00:00
danielr
d8677ba611 fixed ConcurrentModificationException in HttpConnectionInfos 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4690 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-04-13 11:25:41 +00:00
orbiter
c7021c14bb patch for ArrayIndexOutOfBoundsException in BMP parser 
(may occur in case of malformed BMPs)

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4689 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-04-13 03:28:26 +00:00
orbiter
8dd35f74c8 fixed redirect problem (does not work for POST) 
see http://forum.yacy-websuche.de/viewtopic.php?f=6&t=1068&hilit=

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4687 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-04-12 16:35:09 +00:00
orbiter
8313d58ae7 - integrated the collage into the Web Visualization menu 
- added a counter for the public and private queue on the page (testing..)
- fixed wrong public/private categorization

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4686 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-04-12 15:45:57 +00:00
danielr
2617f4dcdb Connections_p.html: better formatting and remove very old entries 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4684 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-04-12 15:19:18 +00:00
orbiter
82bf9ac1c8 - added Collage servlet from datengrab and modified it: 
* all images are queued
* private/public is respected
* inserted into switchboard
* added collageQueue class that stores all the queued images

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4683 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-04-12 13:24:21 +00:00
danielr
959f448e5f - disabled redirects in proxy (so client sees real path) 
- added connection stats (only connections currently in use)
- remove "old" connections (closed or idle for some time)
- synchronized shared parts of proxyHandler


git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4682 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-04-12 11:39:48 +00:00
orbiter
8fe39ebd74 -fixed file transmission with POST. The only usage was in ranking transmission, therefore: 
-fixed ranking transmission

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4681 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-04-12 08:12:51 +00:00
orbiter
82a9861779 fix for last commit 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4680 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-04-11 12:55:43 +00:00
orbiter
5d1fbb25e7 fix for bad deploy: 
- the name of downloaded release files is adopted if the httpc delivers uncompressed tar.gz files (the .gz is removed from the file name)
- the deploy method is able to handle tar-file (not tar.gz-files)

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4679 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-04-11 12:37:17 +00:00
orbiter
202a3adb3e refactoring of HttpClient Writer processes 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4678 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-04-10 22:47:05 +00:00
danielr
8aa9fd8f24 HTTPC with only 1 retry 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4677 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-04-10 16:47:57 +00:00
orbiter
444dce7e81 more performance hacks 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4676 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-04-10 15:28:58 +00:00
orbiter
2c2dcd12a2 - enhanced performance of Eco-Tables: less time-consuming size() - operations 
- will increase speed of indexing and collection.index creation


git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4675 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-04-10 13:24:55 +00:00
orbiter
e356625b22 - refacotring of stream copy handling to support time-consuming operations 
- made usage of BufferedStreams explizit to distinct different copy method in serverFileUtils (byte-by-byte and using an own buffer)
- introduced another timeout setting (java internal property)
- more restrictions to clients accessing a single host (a security setting to prevent DoS by mistake)

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4674 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-04-10 09:53:07 +00:00
danielr
f01c50cf8d Proxy logging error (first step to resolution!?) 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4673 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-04-10 06:56:06 +00:00
orbiter
c3342e1178 - removed class with only one static method 
- removed connection method with too long time-out

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4672 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-04-09 23:35:20 +00:00
orbiter
f97971b63b fixed NPE problems doing a shutdown from command-line 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4671 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-04-09 22:59:17 +00:00
danielr
7a35126e91 http timeouts von alten httpc wieder gesetzt 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4670 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-04-09 11:02:14 +00:00
orbiter
2c1c3bb6eb - some refactoring (sorry Daniel, hab in deinem Code rumgewütet) 
- fixed broken downloads (flush was missing)
- different problem handling when download is corrupted
- different default values in yacy.init

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4669 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-04-08 21:36:33 +00:00
danielr
d96e2badc7 - fixed POST in proxy 
- prepared http connection tracking
- refactoring (mainly moving StreamTools to serverFileUtils)


git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4668 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-04-08 21:17:40 +00:00
orbiter
14404d31a8 - enhanced performance graph (more info) 
- added conditions for rarely used logging lines to prevent unnecessary CPU usage for non-printed info

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4667 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-04-08 14:44:39 +00:00
orbiter
696b8ee3f5 fix for http://forum.yacy-websuche.de/viewtopic.php?p=6806#p6806 
- removed all InputStream.available() because this does not work for files > 2GB
- iterator terminate when a IOException occurs
- added handling of non-executing index.add methods to enhance assert usage
- added index for file indexes > 2GB, to be used in new indexHeap

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4666 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-04-08 11:55:59 +00:00
danielr
94d3d3a86f fixed Proxy (for GET, POST still does not work!) 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4665 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-04-08 09:34:20 +00:00