Commit Graph

2352 Commits

Author SHA1 Message Date
orbiter
e03fcf4627 SSI fix for http://forum.yacy-websuche.de/viewtopic.php?f=6&t=29
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3936 6c8d7289-2bf4-0310-a012-ef5d649a1542
2007-06-29 10:45:13 +00:00
orbiter
1a45ecb356 - fix for http://forum.yacy-websuche.de/viewtopic.php?f=5&t=14&p=137#p137
- fix for missing restart script in ant built target
- removed some more synchronization for size() operations
- removed blocking statement on search page

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3935 6c8d7289-2bf4-0310-a012-ef5d649a1542
2007-06-28 22:06:33 +00:00
orbiter
1fa4feb8e6 added restart button. should work on linux and mac, but was only tested on mac
should of course work on windows as before

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3934 6c8d7289-2bf4-0310-a012-ef5d649a1542
2007-06-28 20:36:15 +00:00
orbiter
f1ed91a8e4 added option to allow/disallow DHT transmission during indexing
see also http://forum.yacy.de/viewtopic.php?f=9&t=8

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3933 6c8d7289-2bf4-0310-a012-ef5d649a1542
2007-06-28 15:25:33 +00:00
orbiter
87afdfc2a7 fix for long waiting time during deletion of processed news
see http://forum.yacy.de/viewtopic.php?f=6&t=6

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3932 6c8d7289-2bf4-0310-a012-ef5d649a1542
2007-06-28 15:03:58 +00:00
orbiter
9bbd39b67c - removed unfinished auto-updater from roland and martin
- added new download-option for releases on the status page
still mising:
- thomas-style restart for linux/mac
- untar/gunzip on shell basis
(comes next)

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3931 6c8d7289-2bf4-0310-a012-ef5d649a1542
2007-06-28 14:52:26 +00:00
orbiter
154ffd7c2c fix for wrong http connection version and SSIs
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3928 6c8d7289-2bf4-0310-a012-ef5d649a1542
2007-06-26 15:06:23 +00:00
orbiter
1782ef57e5 - added SSI parser and include directive for <!--# include virtual="<file>" -->
- added chunked file transfer for non-yacy clients
- SSIs are streamed using chunked transfer, partly delivered pages can be seen in browser before transmission is finished
- added client-side network unit identification
- cleaned up code

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3926 6c8d7289-2bf4-0310-a012-ef5d649a1542
2007-06-26 14:37:10 +00:00
orbiter
6b4cfbd2d6 new network bootsraping method
- no more contact to yacy.net (no remote superseed any more)
- moved superseed file into new network unit definition
- fixed build; includes new network bootstraping files now

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3922 6c8d7289-2bf4-0310-a012-ef5d649a1542
2007-06-25 14:43:57 +00:00
orbiter
0e57a8062b added network definition for different YaCy networks
(needs much more work)

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3919 6c8d7289-2bf4-0310-a012-ef5d649a1542
2007-06-22 14:29:14 +00:00
auron_x
1d41ebf489 *) made age for deletion of too old seeds configurable
*) changed naming-scheme of seed-deletion-properties

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3918 6c8d7289-2bf4-0310-a012-ef5d649a1542
2007-06-22 10:42:29 +00:00
auron_x
52cb3208d0 *) old (lastseen > 7d) peers are now automatically removed from passive and potential seed-dbs
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3917 6c8d7289-2bf4-0310-a012-ef5d649a1542
2007-06-22 09:16:25 +00:00
orbiter
815e3da62f fix for http://www.yacy-forum.de/viewtopic.php?p=37353#37353
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3913 6c8d7289-2bf4-0310-a012-ef5d649a1542
2007-06-18 11:34:30 +00:00
michitux
25529290ca - 2 small changes in documentation
- hopefully fixed logging of GCs (in order to avoid things like "performed necessary GC, freed 18014398509481565 KB (requested/available/average: 4096 / 1631 / 2957 KB)") with the help of KoH


git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3909 6c8d7289-2bf4-0310-a012-ef5d649a1542
2007-06-17 19:32:38 +00:00
low012
c59a7ce5c2 *) hopefully fixed a stupid bug (my fault of course) that sometimes messed up the marking of search words in the snippets (see http://www.yacy-forum.de/viewtopic.php?p=37329#37329)
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3908 6c8d7289-2bf4-0310-a012-ef5d649a1542
2007-06-17 16:29:04 +00:00
orbiter
6518bb6c08 changed release strategy:
we will provide two different releases in the future, one standard release and one 'pro'-release.
the 'pro'-release contains all additional parsers AND has different default performance values.
The pro-version differs therefore from the previous 'all'-version by this default values.
The pro-configuration is automatically choosen if the libx-folder exists. If a version is once initialized, its configuration stays independently from an existing libx folder.
The ant targets had been changed. There are now 3 different targets to create standard and pro-releases, and one target to upgrade:
- dist: creates a standard release (only, no libx target any more)
- distPro: creates a pro-release (includes the libx)
- distExt: creates a libx-release which includes the libx-folder only. It may be used to upgrade from standard to pro
Furthermore, the naming of 'dev'-releases had been removed.

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3902 6c8d7289-2bf4-0310-a012-ef5d649a1542
2007-06-16 14:11:52 +00:00
orbiter
069562a14d fixed problem with re-crawl; replaced error file-db with ram-db
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3900 6c8d7289-2bf4-0310-a012-ef5d649a1542
2007-06-15 23:47:08 +00:00
orbiter
c7a614830a several bugfixes
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3899 6c8d7289-2bf4-0310-a012-ef5d649a1542
2007-06-15 17:45:49 +00:00
allo
465145cb6f revert to insecure, but dau-proof defaults
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3898 6c8d7289-2bf4-0310-a012-ef5d649a1542
2007-06-15 12:56:52 +00:00
allo
7ad11ceaaa security fix for peers without password. allow access only from localhost
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3897 6c8d7289-2bf4-0310-a012-ef5d649a1542
2007-06-15 00:03:44 +00:00
(no author)
2784820ee3 *) moving sleep to a better place
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3895 6c8d7289-2bf4-0310-a012-ef5d649a1542
2007-06-14 16:53:22 +00:00
theli
7a1b811d18 *) bugfix for SocketException:
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3893 6c8d7289-2bf4-0310-a012-ef5d649a1542
2007-06-14 15:58:10 +00:00
orbiter
71fd972ac0 - reduced default search time
- catched case when web structure cannot be painted because of too less data
- better logging when balance fails


git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3892 6c8d7289-2bf4-0310-a012-ef5d649a1542
2007-06-14 15:21:01 +00:00
orbiter
2b937abef1 slighlty different behavior in shutdown sequence for http server threads:
- first close streams
- make pause (that one that was made in httpdFileHandler)
- close sockets

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3890 6c8d7289-2bf4-0310-a012-ef5d649a1542
2007-06-14 11:58:20 +00:00
orbiter
e4aa8f2a08 disabled more sleep(200)
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3889 6c8d7289-2bf4-0310-a012-ef5d649a1542
2007-06-14 11:02:28 +00:00
orbiter
cb38e57622 reduced httpd final waiting time
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3888 6c8d7289-2bf4-0310-a012-ef5d649a1542
2007-06-14 10:28:37 +00:00
orbiter
b4585ad67d im Sommer 2005 wurden die ersten pings zwischen YaCy-Peers ausgetauscht.
Das klappte aber merkwürdigerweise nicht immer. Um das Protokoll zu testen schrieb ich eine einfache message-Funktion, so wie sie heute noch in YaCy drin ist.
Aber auch die Messages funktionierten nicht richtig. Alex und ich haben lange Zeit gesucht, und den Fehler nie gefunden. Es stellte sich heraus das ein Timing-Detail das Problem lösen konnte, die Ursache haben wir bis heute nicht gefunden.
Die Lösung des Problems bestand aus einem kurzen sleep, kurz bevor der httpd Daten zum client zurück geschrieben hat. Das ist natürlich eine fürchterlich schlechte Lösung.
Bis heute war diese Sache im httpd. Mit diesem Commit habe ich den sleep auskommentiert, und es steht zu befürchten das wieder irgendwas nicht geht.
Wenn jetzt das Netz zusammenbricht, keine pings mehr ankommen oder so, war es dieses sleep, das es verhinderte.
Vorschläge willkommen.

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3887 6c8d7289-2bf4-0310-a012-ef5d649a1542
2007-06-14 08:18:10 +00:00
karlchenofhell
e1d809d5f1 - more detailed logging of MEMORY messages
- forced GCs don't contribute to heuristics anymore

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3881 6c8d7289-2bf4-0310-a012-ef5d649a1542
2007-06-13 15:03:56 +00:00
orbiter
9f7765863b bugfix for seed length control routine
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3879 6c8d7289-2bf4-0310-a012-ef5d649a1542
2007-06-13 13:21:19 +00:00
orbiter
0b10ef64ba better server access tracking
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3878 6c8d7289-2bf4-0310-a012-ef5d649a1542
2007-06-13 13:05:51 +00:00
orbiter
4f5496062c protection against too large seeds
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3877 6c8d7289-2bf4-0310-a012-ef5d649a1542
2007-06-12 22:08:33 +00:00
orbiter
684ded0e09 added new news types
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3876 6c8d7289-2bf4-0310-a012-ef5d649a1542
2007-06-12 15:15:24 +00:00
karlchenofhell
669f840eab - added ViewProfile / Impressum (default on) to local peer's robots.txt
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3874 6c8d7289-2bf4-0310-a012-ef5d649a1542
2007-06-12 13:48:27 +00:00
orbiter
d7de0938a6 fix for http://www.yacy-forum.de/viewtopic.php?p=36587#36587
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3870 6c8d7289-2bf4-0310-a012-ef5d649a1542
2007-06-12 08:47:34 +00:00
karlchenofhell
22ee85ca02 - specified exceptions thrown by ResourceInfoFactory and plasmaHTCache.loadResourceInfo()
- caught possible NPE in CacheAdmin_p and added more error-cases
- speeded up deletion of entries in the local crawl queue by crawl profile (it has been noted often that this deletion is slow)
- added a bit javadoc

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3868 6c8d7289-2bf4-0310-a012-ef5d649a1542
2007-06-11 23:33:24 +00:00
orbiter
dfd5e823c3 automatic limitation of web structure host count
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3867 6c8d7289-2bf4-0310-a012-ef5d649a1542
2007-06-11 22:08:10 +00:00
orbiter
8b0aea6910 fixed automatic deletion of too many referenced hosts in web structure
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3866 6c8d7289-2bf4-0310-a012-ef5d649a1542
2007-06-11 21:51:56 +00:00
orbiter
5dd9acc2a7 removed calls to deprecated methods
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3865 6c8d7289-2bf4-0310-a012-ef5d649a1542
2007-06-11 21:33:45 +00:00
orbiter
66ec8b63c1 added a httpd access tracker:
- all requests to the own httdp can now be listed in the access tracker menu
- the search statistics had been renamed to access tracker and extended by this tracker

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3861 6c8d7289-2bf4-0310-a012-ef5d649a1542
2007-06-11 14:05:20 +00:00
karlchenofhell
71ca9aa6d4 - fix for changed blacklist types
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3857 6c8d7289-2bf4-0310-a012-ef5d649a1542
2007-06-11 09:56:16 +00:00
orbiter
9a8a87612d added new qph column to search tracker servlet
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3854 6c8d7289-2bf4-0310-a012-ef5d649a1542
2007-06-10 22:02:17 +00:00
orbiter
e07458bad4 added time-out function to web analysis
the default time-out is 1 second

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3852 6c8d7289-2bf4-0310-a012-ef5d649a1542
2007-06-10 20:00:44 +00:00
hydrox
4a1bc4743a *)News-entries with blacklisted URLs are now ignored
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3849 6c8d7289-2bf4-0310-a012-ef5d649a1542
2007-06-10 08:05:18 +00:00
allo
6074264267 dynamic rights.
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3847 6c8d7289-2bf4-0310-a012-ef5d649a1542
2007-06-09 19:34:09 +00:00
theli
99062c0c9e *) SOAP should support authentication against the user-DB now (requested by KoH)
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3846 6c8d7289-2bf4-0310-a012-ef5d649a1542
2007-06-09 15:49:04 +00:00
theli
339153d40e *) favicons that are specified in the document content via html link-tags
are now detected and displayed on the search page (requested by allo).

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3845 6c8d7289-2bf4-0310-a012-ef5d649a1542
2007-06-09 15:22:37 +00:00
allo
854eb1492f .yacy /.yacyh urls for the feedreader
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3844 6c8d7289-2bf4-0310-a012-ef5d649a1542
2007-06-09 12:56:08 +00:00
allo
7a5b22a0b8 Integration of FeedReader in Bookmarks.
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3841 6c8d7289-2bf4-0310-a012-ef5d649a1542
2007-06-08 23:27:42 +00:00
karlchenofhell
6265d321bd - more constants
- display why global search is not available on search page

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3839 6c8d7289-2bf4-0310-a012-ef5d649a1542
2007-06-08 20:01:16 +00:00
allo
7921f07c9d userDB fix
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3837 6c8d7289-2bf4-0310-a012-ef5d649a1542
2007-06-08 16:11:10 +00:00