Commit Graph

292 Commits

Author SHA1 Message Date
orbiter
ec72387165 added a very early test version of a YaCy gui component.
The gui currently does nothing else than providing a search window that sends the search string to the browser
The gui is started when YaCy is started with the option -g or --gui, like
./startYACY.sh -g
The gui will primary be used to provide a 'real' macintosh version that can be started and operated like any other macintosh application. A special mac application wrapper will follow.

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7021 6c8d7289-2bf4-0310-a012-ef5d649a1542
2010-08-05 10:43:03 +00:00
sixcooler
d88b9606d1 fix for http://forum.yacy-websuche.de/viewtopic.php?f=6&t=2923
+ some client fine tune

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7020 6c8d7289-2bf4-0310-a012-ef5d649a1542
2010-08-04 16:58:33 +00:00
orbiter
6388a58fc7 better memory management and slightly less (in total and temporary) RAM allocation:
- confirm that database objects that are not supposed to grow do not have a index memory management that is designed for growth
- changed index sorting method in such a way that it allocates less objects during quicksort
- database classes classes renaming (shorter, naming addresses that objects hold in RAM)
- added a large number of asserts to check if objects actually take the RAM that they should have


git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7019 6c8d7289-2bf4-0310-a012-ef5d649a1542
2010-08-04 13:33:12 +00:00
orbiter
5924a0d851 - enhanced concurrency in database index access for multicore
- added statistics about database index caches in PerformanceMemory_p.html
- adoped many classes to use the new statistics
- added missing close statements

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7018 6c8d7289-2bf4-0310-a012-ef5d649a1542
2010-08-03 04:58:48 +00:00
orbiter
55a2536bcf enhancement in drawing speed and reduction of object allocation during drawing
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7017 6c8d7289-2bf4-0310-a012-ef5d649a1542
2010-08-03 02:44:08 +00:00
orbiter
9ab06bc333 enhancement in sorting efficiency (database root operation): less object allocation
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7015 6c8d7289-2bf4-0310-a012-ef5d649a1542
2010-08-03 02:42:28 +00:00
sixcooler
39d96abbb5 fix yacyRelease download
(http://forum.yacy-websuche.de/viewtopic.php?f=5&t=2920&p=20545#p20545)
better cookie policy

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7014 6c8d7289-2bf4-0310-a012-ef5d649a1542
2010-08-02 20:13:20 +00:00
sixcooler
349e4dee9d ... migrating to HttpComponents-Client-4.x ...
added cookie policy

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7012 6c8d7289-2bf4-0310-a012-ef5d649a1542
2010-08-02 14:16:44 +00:00
sixcooler
c29f24a519 ... migrating to HttpComponents-Client-4.x ...
- Proxy
- Release-download

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7011 6c8d7289-2bf4-0310-a012-ef5d649a1542
2010-08-01 22:35:11 +00:00
orbiter
d5c65b17a6 added another network activity visualization: show strong query activity as radiation around peer
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7006 6c8d7289-2bf4-0310-a012-ef5d649a1542
2010-07-28 11:40:58 +00:00
orbiter
989948e1a9 fixed generic image parser
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7005 6c8d7289-2bf4-0310-a012-ef5d649a1542
2010-07-27 07:13:15 +00:00
orbiter
e1015ead2c static access to constants
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7004 6c8d7289-2bf4-0310-a012-ef5d649a1542
2010-07-27 06:52:58 +00:00
orbiter
27d8a8b53e removed wrong com.sun.codec class access in generic image parser
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7003 6c8d7289-2bf4-0310-a012-ef5d649a1542
2010-07-27 06:49:09 +00:00
orbiter
bbf887d879 added generics to UPnP classes
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7002 6c8d7289-2bf4-0310-a012-ef5d649a1542
2010-07-27 06:48:01 +00:00
sixcooler
15e8c13526 ... migrating to HttpComponents-Client-4.x ...
(gzip decompression, httploader, robots, ...)

+ enable proxy-crawling while log is fine

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7001 6c8d7289-2bf4-0310-a012-ef5d649a1542
2010-07-27 01:16:26 +00:00
mikeworks
b12db14b9f Added Generics to new net.yacy.upnp.* classes to eliminate compiler warnings
Added @Deprecated for deprecated functions getIPDevices and getPPPDevices in class InternetGatewayDevice
Changed debug statement in Domains.java and corrected filename in comments header

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6993 6c8d7289-2bf4-0310-a012-ef5d649a1542
2010-07-24 13:48:45 +00:00
sixcooler
b7102eff92 ... migrating to HttpComponents-Client-4.x ...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6989 6c8d7289-2bf4-0310-a012-ef5d649a1542
2010-07-22 23:08:37 +00:00
mikeworks
572e429eff - fixes UPnP not working discussion on forum: http://forum.yacy-websuche.de/viewtopic.php?f=5&t=2881
SVN 6987 fixed net.yacy.upnp.devices.UPNPRootDevice for usage with JxPath > 1.3 by using a default namespace (xmlns="urn:schemas-upnp-org:device-1-0")
This commit now fixes the same problem for net.yacy.upnp.devices.UPNPService with default namespace (xmlns="urn:schemas-upnp-org:service-1-0")

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6988 6c8d7289-2bf4-0310-a012-ef5d649a1542
2010-07-22 19:13:37 +00:00
mikeworks
2a20282505 git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6987 6c8d7289-2bf4-0310-a012-ef5d649a1542 2010-07-22 19:07:02 +00:00
lotus
965aa97993 including sbbi upnplib as source again
http://www.sbbi.net/site/upnp/index.html

renamed package to yacy
all options are also named "yacy" instead of "sbbi"

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6986 6c8d7289-2bf4-0310-a012-ef5d649a1542
2010-07-22 18:02:16 +00:00
orbiter
60caade056 removed debug output
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6984 6c8d7289-2bf4-0310-a012-ef5d649a1542
2010-07-22 07:59:22 +00:00
sixcooler
52718e6dcb ... migrating to HttpComponents-Client-4.x ...
monitoring: replaced unused 'idletime' by uploading bytes
added some kind of 'upload-throttling' at dht-out :-)

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6983 6c8d7289-2bf4-0310-a012-ef5d649a1542
2010-07-22 00:51:41 +00:00
sixcooler
5fa8038f10 ... migrating to HttpComponents-Client-4.x ...
monitoring and first try to use remoteProxy

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6979 6c8d7289-2bf4-0310-a012-ef5d649a1542
2010-07-20 01:14:28 +00:00
orbiter
dec1419bc3 ;-)
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6978 6c8d7289-2bf4-0310-a012-ef5d649a1542
2010-07-18 20:18:32 +00:00
orbiter
22dbbcfa56 better (and corrected) recognition of intranet and internet-addresses. This corrects the isLocal property that is used by network definitions to restrict index ranges to local and global addresses. Address locations (intranet or internet) had been partly identified by the top level domain of the host address. Since intranet addresses can also be addressed using a host name that is in a country domain it is necessary to do a dns resolving for each check. The check is supported by a local dns cache so the intranet/internet check should not affect network traffic too much. To ensure that the cache works properly the cache class was upgraded to better concurrency data structures.
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6977 6c8d7289-2bf4-0310-a012-ef5d649a1542
2010-07-18 20:14:20 +00:00
orbiter
8674a65488 removed override directive which caused a compile error in eclipse helios
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6974 6c8d7289-2bf4-0310-a012-ef5d649a1542
2010-07-18 18:37:20 +00:00
low012
dc5f0e357c *) fixed SVN properties
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6972 6c8d7289-2bf4-0310-a012-ef5d649a1542
2010-07-18 10:02:03 +00:00
low012
01d6b952f0 *) minor changes for easier to read code, no functional changes
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6971 6c8d7289-2bf4-0310-a012-ef5d649a1542
2010-07-18 10:00:43 +00:00
sixcooler
0e56d29335 ... migrating to HttpComponents-Client-4.x ...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6968 6c8d7289-2bf4-0310-a012-ef5d649a1542
2010-07-15 00:59:53 +00:00
sixcooler
2ad5829b26 correct Timeoutparamter at HttpComponents-Client-4.x
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6967 6c8d7289-2bf4-0310-a012-ef5d649a1542
2010-07-14 02:47:40 +00:00
sixcooler
e1316d12d0 ... migrating to HttpComponents-Client-4.x ...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6966 6c8d7289-2bf4-0310-a012-ef5d649a1542
2010-07-13 22:10:24 +00:00
sixcooler
c5c67f0504 start migrating to HttpComponents-Client-4.x
see http://forum.yacy-websuche.de/viewtopic.php?f=5&t=2872

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6965 6c8d7289-2bf4-0310-a012-ef5d649a1542
2010-07-12 23:07:05 +00:00
orbiter
25024d6ab2 fix for problen when accessing the metadata index. The index was not available for all peers with no RAM table copy.
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6957 6c8d7289-2bf4-0310-a012-ef5d649a1542
2010-06-30 07:22:50 +00:00
orbiter
b6fb239e74 redesign of parser interface:
some file types are containers for several files. These containers had been parsed in such a way that the set of resulting parsed content was merged into one single document before parsing. Using this parser infrastructure it is not possible to parse document containers that contain individual files. An example is a rss file where the rss messages can be treated as individual documents with their own url reference. Another example is a surrogate file which was treated with a special operation outside of the parser infrastructure.
This commit introduces a redesigned parser interface and a new abstract parser implementation. The new parser interface has now only one entry point and returns always a set of parsed documents. In case of single documents the parser method returns a set of one documents.
To be compliant with the new interface, the zip and tar parser had been also completely redesigned. All parsers are now much more simple and cleaner in its structure. The switchboard operations had been extended to operate with sets of parsed files, not single parsed files.
additionally, parsing of jar manifest files had been added.

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6955 6c8d7289-2bf4-0310-a012-ef5d649a1542
2010-06-29 19:20:45 +00:00
low012
d4851441b0 *) Added Android packages to parser in order to be able to create a decentralized search for direct downloads of Android apps.
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6953 6c8d7289-2bf4-0310-a012-ef5d649a1542
2010-06-28 20:41:08 +00:00
orbiter
150cf42a1b migrated all my LGPL 3 -licensed files to the LGPL 2.1 because LGPL 3 is not compatible to the GPL 2
see http://www.gnu.org/licenses/license-list.html for explanation
Since (as far as I know) nobody else has ever contributed to these files I may be allowed to just apply an older license.
You may consider this as a dual-licensing and may use and optionally replicate the older files under GPL 3.

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6952 6c8d7289-2bf4-0310-a012-ef5d649a1542
2010-06-28 16:25:14 +00:00
orbiter
5d00888c95 - added animated visualization for DHT-in and DHT-out in network graphic
- found and fixed a possible memory leak in YaCy internal RSS feed system
- some refactoring in RSS feed mechanisms to make this possible

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6950 6c8d7289-2bf4-0310-a012-ef5d649a1542
2010-06-27 10:45:20 +00:00
orbiter
bf25407fdd added peer hash to internal RSSFeed. The hash will be used to display news activities in the network graphic.
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6949 6c8d7289-2bf4-0310-a012-ef5d649a1542
2010-06-26 23:10:57 +00:00
orbiter
1557e0f2d0 - some refactoring for internal RSSFeed (protocol of all actions as seen on status page)
- added dht-out to internal RSSFeed (you can see now messages about distributed indexes on status page)

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6948 6c8d7289-2bf4-0310-a012-ef5d649a1542
2010-06-26 22:39:27 +00:00
orbiter
5a4684f21f allow words with length >= 2 (you can't search for 'wm' with 3-letter words...)
lets try that. If we run into a memory problem because of too many 2-letter-words, then we must introduce whitelists for 2-letter words.

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6947 6c8d7289-2bf4-0310-a012-ef5d649a1542
2010-06-26 16:31:26 +00:00
orbiter
37b8827a7a - removed the UPnP library sources from sbbi and added the jar library again. The library was included to get support for fedora releases, but after this time the fact that the sbbi cannot be part of fedora should be re-discussed. If this will still not be possible, then we may integrate the sbbi UPnP package using reflection.
- cleaned uo the code. The new eclipse helios provided new warnings for dead code. This change cleans up most of these warnings

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6945 6c8d7289-2bf4-0310-a012-ef5d649a1542
2010-06-26 10:32:47 +00:00
orbiter
103c848af8 enhancements in image drawing speed
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6942 6c8d7289-2bf4-0310-a012-ef5d649a1542
2010-06-24 13:20:45 +00:00
orbiter
777195e8d1 more abstraction for access of LoaderDispatcher and cache
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6937 6c8d7289-2bf4-0310-a012-ef5d649a1542
2010-06-22 12:28:53 +00:00
orbiter
7bcfa033c9 more abstraction of the htcache when using the LoaderDispatcher:
a cache access shall not made directly to the cache any more, all loading attempts shall use the LoaderDispatcher.
To control the usage of the cache, a enum instance from CrawlProfile.CacheStrategy shall be used.
Some direct loading methods without the usage of a cache strategy have been removed. This affects also the verify-option
of the yacysearch servlet. If there is a 'verify=false' now after this commit this does not necessarily mean that no snippets
are generated. Instead, all snippets that can be retrieved using the cache only are presented. This still means that the search hit was not verified because the snippet was generated using the cache. If a cache-based generation of snippets is not possible, then the verify=false causes that the link is not rejected.

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6936 6c8d7289-2bf4-0310-a012-ef5d649a1542
2010-06-21 14:54:54 +00:00
orbiter
7e2d6fac12 patch for bad values during local search join
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6934 6c8d7289-2bf4-0310-a012-ef5d649a1542
2010-06-20 00:31:00 +00:00
orbiter
986d4f34d9 added a consistency check for new queues
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6931 6c8d7289-2bf4-0310-a012-ef5d649a1542
2010-06-18 18:59:42 +00:00
orbiter
73f03e05ee fixed a bug in snippet fetch strategy: cache only does not help if resource can only be found in web
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6930 6c8d7289-2bf4-0310-a012-ef5d649a1542
2010-06-18 15:25:25 +00:00
orbiter
fbf021bb50 redesign of index abstract processing - currently disabled until enough peers have fix in SVN 6928
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6929 6c8d7289-2bf4-0310-a012-ef5d649a1542
2010-06-18 09:44:21 +00:00
orbiter
87087f12fe - scanned remote search process and enhanced some data structure and synchronizations here and there
- removed concurrency overhead for small number of index normalizations as it happens during remote search
- removed 'load only parseable' constraint for snippet fetch because some resources may not have any url file extension and these had therefore not been parseable and searcheable since they may become parseable after loading when their mime type is known
- this partly fixes some problems with http://forum.yacy-websuche.de/viewtopic.php?p=20300#p20300 but more changes are necessary to get all expected search results

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6926 6c8d7289-2bf4-0310-a012-ef5d649a1542
2010-06-17 11:59:40 +00:00
orbiter
7ddb70e7c6 new license for ai.greedy component: LGPL (nobody else than me modified that code)
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6925 6c8d7289-2bf4-0310-a012-ef5d649a1542
2010-06-16 22:16:03 +00:00