Commit Graph

454 Commits

Author SHA1 Message Date
orbiter
a4e8ad95ab enhancements to news and switchboard queue processing
removed direct access and replaced by iteration

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3961 6c8d7289-2bf4-0310-a012-ef5d649a1542
2007-07-13 13:00:18 +00:00
orbiter
a45216b479 fix to prevent bad-formed news messages
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3960 6c8d7289-2bf4-0310-a012-ef5d649a1542
2007-07-13 09:41:55 +00:00
orbiter
bec4dbc753 added options and execution methods for automated updates
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3959 6c8d7289-2bf4-0310-a012-ef5d649a1542
2007-07-12 16:23:33 +00:00
orbiter
208b5297f1 enhanced handling of news records:
result is a speedup of Surftips, Supporter, and Network page

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3954 6c8d7289-2bf4-0310-a012-ef5d649a1542
2007-07-05 22:56:37 +00:00
orbiter
3421c64d26 implemented update function:
after downloading a release using the download button on the status page
the user can choose any of the downloaded versions for a update.
this enables also a downgrade to a older version.
when the update button is pushed, yacy terminates, installes the choosen version
and restarts

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3948 6c8d7289-2bf4-0310-a012-ef5d649a1542
2007-07-02 15:16:05 +00:00
orbiter
f1ed91a8e4 added option to allow/disallow DHT transmission during indexing
see also http://forum.yacy.de/viewtopic.php?f=9&t=8

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3933 6c8d7289-2bf4-0310-a012-ef5d649a1542
2007-06-28 15:25:33 +00:00
orbiter
9bbd39b67c - removed unfinished auto-updater from roland and martin
- added new download-option for releases on the status page
still mising:
- thomas-style restart for linux/mac
- untar/gunzip on shell basis
(comes next)

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3931 6c8d7289-2bf4-0310-a012-ef5d649a1542
2007-06-28 14:52:26 +00:00
orbiter
1782ef57e5 - added SSI parser and include directive for <!--# include virtual="<file>" -->
- added chunked file transfer for non-yacy clients
- SSIs are streamed using chunked transfer, partly delivered pages can be seen in browser before transmission is finished
- added client-side network unit identification
- cleaned up code

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3926 6c8d7289-2bf4-0310-a012-ef5d649a1542
2007-06-26 14:37:10 +00:00
orbiter
0e57a8062b added network definition for different YaCy networks
(needs much more work)

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3919 6c8d7289-2bf4-0310-a012-ef5d649a1542
2007-06-22 14:29:14 +00:00
auron_x
1d41ebf489 *) made age for deletion of too old seeds configurable
*) changed naming-scheme of seed-deletion-properties

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3918 6c8d7289-2bf4-0310-a012-ef5d649a1542
2007-06-22 10:42:29 +00:00
auron_x
52cb3208d0 *) old (lastseen > 7d) peers are now automatically removed from passive and potential seed-dbs
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3917 6c8d7289-2bf4-0310-a012-ef5d649a1542
2007-06-22 09:16:25 +00:00
orbiter
6518bb6c08 changed release strategy:
we will provide two different releases in the future, one standard release and one 'pro'-release.
the 'pro'-release contains all additional parsers AND has different default performance values.
The pro-version differs therefore from the previous 'all'-version by this default values.
The pro-configuration is automatically choosen if the libx-folder exists. If a version is once initialized, its configuration stays independently from an existing libx folder.
The ant targets had been changed. There are now 3 different targets to create standard and pro-releases, and one target to upgrade:
- dist: creates a standard release (only, no libx target any more)
- distPro: creates a pro-release (includes the libx)
- distExt: creates a libx-release which includes the libx-folder only. It may be used to upgrade from standard to pro
Furthermore, the naming of 'dev'-releases had been removed.

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3902 6c8d7289-2bf4-0310-a012-ef5d649a1542
2007-06-16 14:11:52 +00:00
orbiter
069562a14d fixed problem with re-crawl; replaced error file-db with ram-db
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3900 6c8d7289-2bf4-0310-a012-ef5d649a1542
2007-06-15 23:47:08 +00:00
orbiter
c7a614830a several bugfixes
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3899 6c8d7289-2bf4-0310-a012-ef5d649a1542
2007-06-15 17:45:49 +00:00
allo
465145cb6f revert to insecure, but dau-proof defaults
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3898 6c8d7289-2bf4-0310-a012-ef5d649a1542
2007-06-15 12:56:52 +00:00
allo
7ad11ceaaa security fix for peers without password. allow access only from localhost
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3897 6c8d7289-2bf4-0310-a012-ef5d649a1542
2007-06-15 00:03:44 +00:00
orbiter
71fd972ac0 - reduced default search time
- catched case when web structure cannot be painted because of too less data
- better logging when balance fails


git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3892 6c8d7289-2bf4-0310-a012-ef5d649a1542
2007-06-14 15:21:01 +00:00
orbiter
684ded0e09 added new news types
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3876 6c8d7289-2bf4-0310-a012-ef5d649a1542
2007-06-12 15:15:24 +00:00
karlchenofhell
22ee85ca02 - specified exceptions thrown by ResourceInfoFactory and plasmaHTCache.loadResourceInfo()
- caught possible NPE in CacheAdmin_p and added more error-cases
- speeded up deletion of entries in the local crawl queue by crawl profile (it has been noted often that this deletion is slow)
- added a bit javadoc

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3868 6c8d7289-2bf4-0310-a012-ef5d649a1542
2007-06-11 23:33:24 +00:00
orbiter
9a8a87612d added new qph column to search tracker servlet
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3854 6c8d7289-2bf4-0310-a012-ef5d649a1542
2007-06-10 22:02:17 +00:00
karlchenofhell
6265d321bd - more constants
- display why global search is not available on search page

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3839 6c8d7289-2bf4-0310-a012-ef5d649a1542
2007-06-08 20:01:16 +00:00
(no author)
94cc9f05f5 *) Improvements for restart via update wrapper
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3785 6c8d7289-2bf4-0310-a012-ef5d649a1542
2007-06-02 15:25:13 +00:00
orbiter
33ad0c8246 added a web structure computation and logging:
- all web page parsing operations will now increase a web structure file
- the file is computed in memory and dumped at shutdown-time to PLASMASB/webStructure.map in readable form (not a database)
- the file can be used externally to analyse the link structure of the crawled pages
- the web structure can also be retrieved using a xml-interface at http://localhost:8080/xml/webstructure.xml
- the short-term purpose is the computation of a link-graph image (before linuxtag!)
- a long-term purpose could be a decentralized computation of the citation rank



git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3746 6c8d7289-2bf4-0310-a012-ef5d649a1542
2007-05-22 08:13:48 +00:00
karlchenofhell
7904175338 - sorry for typos
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3743 6c8d7289-2bf4-0310-a012-ef5d649a1542
2007-05-20 16:22:46 +00:00
karlchenofhell
baa9402b97 - wiki-parser is now configurable via the config setting wikiParser.class which holds the class-name for the parser to use
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3742 6c8d7289-2bf4-0310-a012-ef5d649a1542
2007-05-20 16:19:25 +00:00
orbiter
b05e2314cf another dht selection fix
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3725 6c8d7289-2bf4-0310-a012-ef5d649a1542
2007-05-14 12:52:39 +00:00
orbiter
578c2ef130 release 0.52
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3715 6c8d7289-2bf4-0310-a012-ef5d649a1542
2007-05-11 22:12:29 +00:00
orbiter
46367afaaa update of memory-protection values
see http://www.yacy-forum.de/viewtopic.php?p=35539#35539

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3709 6c8d7289-2bf4-0310-a012-ef5d649a1542
2007-05-11 18:02:48 +00:00
rramthun
ea87fe5d78 *) Updated German translation
*) Changed "Lost Handle" error to warning (masses of it if deleting crawl-profile)
*) Removed unnecessary code from Windows script 

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3708 6c8d7289-2bf4-0310-a012-ef5d649a1542
2007-05-11 17:48:22 +00:00
orbiter
e602436fda fixed problem with cluster routing
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3684 6c8d7289-2bf4-0310-a012-ef5d649a1542
2007-05-07 20:48:24 +00:00
theli
74dd6cac95 *) signal yacy shutdown to updater
*) some javadoc added

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3658 6c8d7289-2bf4-0310-a012-ef5d649a1542
2007-05-05 16:20:01 +00:00
theli
43748f87fb *) changes required for the uploader
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3655 6c8d7289-2bf4-0310-a012-ef5d649a1542
2007-05-05 15:41:05 +00:00
orbiter
7cf8981a98 - added debugging code for wrong DHT target iterator
- restricted distance constraint from 0.4 to 0.2

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3644 6c8d7289-2bf4-0310-a012-ef5d649a1542
2007-05-03 22:57:55 +00:00
orbiter
dd44a1394f disabled automatic performance setting change
- during crawl start
- each indexing cycle
- for delay values
- for short memory cycles

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3634 6c8d7289-2bf4-0310-a012-ef5d649a1542
2007-05-02 15:39:27 +00:00
orbiter
06b6e35484 fix for a null pointer exception if clusters are not defined
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3632 6c8d7289-2bf4-0310-a012-ef5d649a1542
2007-05-02 12:26:29 +00:00
orbiter
81844e85b2 - fixed more cluster routing problems
- fixed a problem in remote search when balancer caused shift process to wait too long

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3627 6c8d7289-2bf4-0310-a012-ef5d649a1542
2007-04-30 00:39:53 +00:00
orbiter
304ed3f4d2 fix for remote crawl requests in clusters
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3626 6c8d7289-2bf4-0310-a012-ef5d649a1542
2007-04-29 22:52:07 +00:00
orbiter
e48189c710 enhanced cluster routing
- cluster definitions can now contain an addition for local ip addresses
- cluster-cluster communication uses the local ip address instead the global address, if one is given

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3624 6c8d7289-2bf4-0310-a012-ef5d649a1542
2007-04-29 22:05:34 +00:00
orbiter
485bf1ea83 bugfix for robinson/remote crawl bug
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3614 6c8d7289-2bf4-0310-a012-ef5d649a1542
2007-04-27 21:35:43 +00:00
orbiter
e192f616a2 collection of small bugfixes
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3600 6c8d7289-2bf4-0310-a012-ef5d649a1542
2007-04-26 14:28:57 +00:00
orbiter
f8de19fb2f robinson cluster: added client-side protocol implementation
- the network configuration page shows a new option: robinson clusters
- when a global search is made, all robinson peers are excluded, but:
- robinson peers/clusters that provide peer tags and where search words match
  such tags, they are included in global search. Therefore, robinson peers/clusters
  support the global yacy network with their indexes, without doin DHT-exchange


git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3598 6c8d7289-2bf4-0310-a012-ef5d649a1542
2007-04-26 09:51:51 +00:00
orbiter
657585fe0d network functions for robinson peers: server-side protection
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3591 6c8d7289-2bf4-0310-a012-ef5d649a1542
2007-04-24 15:11:12 +00:00
orbiter
89c1511738 - added new Network Configuration menu, can be found in basic settings
- new cluster functions will be available in this menu, but currently not enabled,
  because corresponding interface methods are not ready yet
- shifted remote crawl settings to new network configuration menu
- shifted DHT distribution/receive to the new network configuration menu
- adopted some string constants
- added cluster configuration settings to yacy.init


git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3589 6c8d7289-2bf4-0310-a012-ef5d649a1542
2007-04-23 20:47:07 +00:00
orbiter
ca79362b9d disabling auto-setting of remote crawl performance
see also http://www.yacy-forum.de/viewtopic.php?t=3849

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3577 6c8d7289-2bf4-0310-a012-ef5d649a1542
2007-04-18 18:36:06 +00:00
michitux
4990909178 Some bugfixes, new layout/style for image search results:
* removed divide by zero bug when 20_dhtdistribution_busysleep is 0
 * replaced German comment with wrong charset in source/de/anomic/plasma/plasmaCrawlBalancer.java by an English one
 * replaced the table-fix for floating behind snipped images by a br with clear
 * removed unnecessary old xhtml-files (were not in use, they were created when we weren't having xhtml for testing)
 * new layout for image-search results: replaced the old one with spans and tables inside (not valid) with new divs, now each image snippet container has the same size
TODO:
 * the ids of the snippetLoading-divs aren't valid because ids must start with an alphabetic letter or an underscore, they have to be prefixed
 * in the returned snippet-xml is an unresolved pattern for status (the status is only set for text snippets)

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3566 6c8d7289-2bf4-0310-a012-ef5d649a1542
2007-04-12 18:21:17 +00:00
orbiter
78d04bcbcf fixed bug in search statistics
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3562 6c8d7289-2bf4-0310-a012-ef5d649a1542
2007-04-12 09:24:56 +00:00
orbiter
b79b4082e2 completed search exclusion:
- exclusion on index-level (not only from search snippets)
- exclusion hand-over at remote search protocol

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3556 6c8d7289-2bf4-0310-a012-ef5d649a1542
2007-04-10 12:27:03 +00:00
orbiter
06a7978730 moved url pattern matching for search to better place
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3550 6c8d7289-2bf4-0310-a012-ef5d649a1542
2007-04-05 15:08:14 +00:00
orbiter
159bd0cab5 diverses; b.o. fix for http://www.yacy-forum.de/viewtopic.php?p=33914#33914
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3549 6c8d7289-2bf4-0310-a012-ef5d649a1542
2007-04-05 14:58:29 +00:00
orbiter
40c14a4f0e - better implementation of search query properties
- basic protection against start-up problems when database files are corrupted
- auto-delete of not-critical databases during startup when load error occurs
- on-the-fly reset option for all database tables
- automatic on-the-fly reset for seed tables during enumeration exceptions

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3547 6c8d7289-2bf4-0310-a012-ef5d649a1542
2007-04-05 10:14:48 +00:00