Commit Graph

2930 Commits

Author SHA1 Message Date
orbiter
cddf884bc4 Merge branch 'master' of git@gitorious.org:yacy/rc1.git 2014-10-07 19:27:14 +02:00
Michael Peter Christen
460858fb22 more ipv6 fixes 2014-10-07 18:53:23 +02:00
Michael Peter Christen
5cef88a315 argh.. adding missing java class for latest audio feature 2014-10-07 18:32:39 +02:00
Michael Peter Christen
74957f3760 Merge branch 'master' of ssh://git@gitorious.org/yacy/rc1.git 2014-10-07 17:51:18 +02:00
Michael Peter Christen
2a052f446a Added an experimental audio feedback system.
This is the first element of a new 'decoration' component which may hold
switches for different external appearance parameters.
The first switch in that context is decoration.audio (as usual in
yacy.init). This value is set to false by default, that means the audio
feedback element is switched off by default. To switch it on, set
decoration.audio = true (using /ConfigProperties_p.html). You will then
hear sounds for the following events:
- remote searches
- incoming dht transmissions
- new documents from the crawler
Sound clips are stored in htroot/env/soundclips/ which is done so
because a future implementation will read these files using the http
client and with configurable urls which will make it very easy for the
user to replace the given sounds with own sounds.
2014-10-07 17:51:07 +02:00
Marc Nause
1e6e69bc40 Finished implementation of UPNP:
*) will try other ports if YaCy standard ports are not available
*) distinguish between internal and external port (not sure if this
works 100%)

Still to add: propery in config to enter own external port (in case of
manually configured NAT)
2014-10-07 13:10:06 +02:00
Michael Peter Christen
d0358e568b Merge branch 'master' of ssh://git@gitorious.org/yacy/rc1.git 2014-10-06 17:44:38 +02:00
Michael Peter Christen
e1bc768f9d more IPv6 bugfixes 2014-10-06 17:44:27 +02:00
reger
59c6532a65 add link extraction to pdfParser
this extracts clickable links in pdf and adds it to the list of links

include a test case for this function

this is the corrected comment for commit:
aa2e15d846
2014-10-06 04:51:31 +02:00
reger
aa2e15d846 allow url parameter in worktable apicall
allow url=wwwl?param=a&param=b (with ?, & encoded)
fix:  http://mantis.tokeek.de/view.php?id=100

fix double adding of  '&' in MultiProtocolURL.escape()
2014-10-05 20:05:03 +02:00
orbiter
f3a12801f0 Merge branch 'master' of git@gitorious.org:yacy/rc1.git 2014-10-05 14:50:35 +02:00
orbiter
d93325a578 lazy handling of process_sxt field (part of postprocessing) 2014-10-05 14:50:22 +02:00
Michael Peter Christen
b31db00010 toString fixes 2014-10-05 11:03:57 +02:00
Michael Peter Christen
961f06c0b6 Merge branch 'master' of ssh://git@gitorious.org/yacy/rc1.git 2014-10-05 01:25:12 +02:00
reger
209e0f2fe8 allow url parameter in worktable apicall
allow url=wwwl?param=a&param=b (with ?, & encoded)
fix:  http://mantis.tokeek.de/view.php?id=100

fix double adding of  '&' in MultiProtocolURL.escape()
2014-10-04 04:11:48 +02:00
reger
b5ca20de15 preserve content_type (mime) if supplied in preference of construct in from file type.
(this eventually can benefit image search by using mime only)

reduce redundant field assignment for Solrdocuments created from URIMetadataNode (URIMetadataNode = SolrDocument with partially assigned fields)
2014-10-03 22:08:07 +02:00
reger
fe9f1c594e fix char encoding parameter in UrlProxy 2014-10-03 08:51:23 +02:00
reger
b0c87d8240 fix image search expand box, cut-off of 2nd capture line height
tested with IE11 and Firefox 32 (change worked for both to show 2nd line without cutting off height)

+fix charset parameter in metadataImageParser
+update start errMsgTxt to "java 1.7"
2014-10-03 01:43:05 +02:00
Michael Peter Christen
2c2ed8bf4e typo in javadoc 2014-10-02 09:38:06 +02:00
Michael Peter Christen
528f583d72 ipv6 fixes 2014-10-01 15:32:10 +02:00
Michael Peter Christen
6ee5b4352d Merge branch 'master' of ssh://git@gitorious.org/yacy/rc1.git 2014-10-01 10:21:13 +02:00
Michael Peter Christen
247e626083 IPv6 host parsing bugfixes 2014-10-01 10:21:03 +02:00
reger
fb1fcc2b03 handle noarchive tag, skip writing page to cache
http://mantis.tokeek.de/view.php?id=44
2014-10-01 04:35:34 +02:00
Michael Peter Christen
fe917deb2d when pinging other peers, be able to select the right IP option 2014-10-01 03:47:57 +02:00
Michael Peter Christen
65e6ae52fb IPv6-enhanced Network monitoring page 2014-10-01 03:10:39 +02:00
Michael Peter Christen
3073c69aee Merge branch 'master' of ssh://git@gitorious.org/yacy/rc1.git 2014-09-30 14:54:06 +02:00
Michael Peter Christen
6491270b3a large IPv6 redesign of peer ping methods!
removed preferred IPv4 in start options and added a new field IP6 in
peer seeds which will contain one or more IPv6 addresses. Now every peer
has one or more IP addresses assigned, even several IPv6 addresses are
possible. The peer-ping process must check all given and possible IP
addresses for a backping and return the one IP which was successful when
pinging the peer. The ping-ing peer must be able to recognize which of
the given IPs are available for outside access of the peer and store
this accordingly. If only one IPv6 address is available and no IPv4,
then the IPv6 is stored in the old IP field of the seed DNA.
Many methods in Seed.java are now marked as @deprecated because they had
been used for a single IP only. There is still a large construction site
left in YaCy now where all these deprecated methods must be replaced
with new method calls. The 'extra'-IPs, used by cluster assignment had
been removed since that can be replaced with IPv6 usage in p2p clusters.
All clusters must now use IPv6 if they want an intranet-routing.
2014-09-30 14:53:52 +02:00
reger
eaccce3467 added metadataImageParser for tif and psd (Photoshop) images.
This is a modified genericImageParser adding tif (and psd) support even if java ImageIO plugin for tif is not installed in JDK.
Adds just tif and psd to the available parsers.
Uses the same library to extract metadata, so could eventually be merged with genericImageParser.
All detected metadata are added to the parsed document (potentially some more as with genericImageParser)
2014-09-30 05:04:47 +02:00
reger
a69f5358ff use javax ImageIO getReader to add supported image extension/mime
genericImageParser uses javax ImageIO, supported images depend on available plugins for ImageIO package (this is JDK installation specific). Jpeg, png and gif are availabel by default. Tif and others only on avalable plugin (in classpath).
Add supported image type dynamically on startup.
2014-09-29 07:42:51 +02:00
reger
8b1ce49ee6 remove unused variable timeout 2014-09-29 02:24:29 +02:00
reger
48aed15c48 skip loader wait cycle on concurrent access in nocache configuration.
In nocache config resource is loaded online, leaving no benefit to wait for a faster cache hit.
2014-09-26 23:49:10 +02:00
Michael Peter Christen
67cd4c37bd activated the new apk parser which was already ready but not included in
the parser initialization. To make the apk parser usable, the handling
of application type links had to be modified. Now all documents which
have not a parser attached are placed to the noload-queue while all
other documents are parsed using the associated parser class. This may
have side-Effects on other parsers and the display of different file
classes (images, apps, videos).
2014-09-24 13:32:58 +02:00
orbiter
a922b122a3 added a hack to forward solr search results from an external attached
solr to the YaCy built-in solr search servlet. Its not complete and not
fully correct (there is still a utf8 encoding problem) but it is a way
to get easily requests forwarded through YaCy to an external Solr.
2014-09-22 15:28:54 +02:00
Michael Peter Christen
025516f682 fix for crawl limit for number of pages fail 2014-09-20 13:06:46 +02:00
Michael Peter Christen
2645dc816a added warning for not well-formed postprocessing queries 2014-09-18 14:36:57 +02:00
Michael Peter Christen
437ce3b8a0 added internal api for partial updates to Solr 2014-09-18 14:26:45 +02:00
orbiter
3ac31614a3 added option to reverse-sort YaCy tables (internal API change only) 2014-09-18 11:11:09 +02:00
Michael Peter Christen
6d3d4c4ea6 changed the concurrent enumeration of query results in such a way that
it is now possible to get the results in two steps:
- first retrieve all IDs as given for a query
- then retieve each document individually

This was necessary for very large result sets where a query may run for
hours and is possibly terminated by a solr-internal timeout. This occurs
regulary during postprocessing and therefore this commit may fix
unwanted postprocessing terminations.
2014-09-17 13:58:55 +02:00
Michael Peter Christen
ad35d9294f added a 'stats' table which records some peer statistics twice every
hour. The table can be shown with
http://localhost:8090/Tables_p.html?table=stats

The entries have the following meaning: 
aM: activeLastMonth
aW: activeLastWeek
aD: activeLastDay
aH: activeLastHour
cC: countConnected (Active Senior)
cD: countDisconnected (Passive Senior)
cP: countPotential (Junior)
cR: count of the RWI entries
cI: size of the index (number of documents)

The entry keys are abbreviated to reduce the space in the table as the
name is written again for every row.

This is the beginning of a 'yacystats' micro-alternative als built-in
function in YaCy. Graphics may follow after some time if enough test
data is available.
2014-09-17 12:54:50 +02:00
reger
8284ea751a catch TimeoutException during ping and do not delete yacy.conf during prereadconfigfile
found a situation after crash (reboot) with existing running semaphore but YaCy not running.
Ping generated exception which finally deleted the conf file (during pre-read procedure)
- change to ping (catch exception solved it)
- additionally removed delete yacy.conf file (if needed we need to make a backup)
2014-09-16 23:14:13 +02:00
reger
ffa7c7116f better fix for NPE in image search
replace 8931e14514
2014-09-16 16:43:17 +02:00
Michael Peter Christen
759e7d9538 fix for http://forum.yacy-websuche.de/viewtopic.php?p=30720#p30720 2014-09-16 14:53:30 +02:00
Michael Peter Christen
bf18a39d0e replaced warning with info 2014-09-16 14:41:04 +02:00
Michael Peter Christen
f1032fb8fe more enhancements to image search in case that a restriction to a single
domain is done
2014-09-16 13:41:01 +02:00
Michael Peter Christen
475125f9d7 hack to get more results when doing a remote site search 2014-09-16 00:13:26 +02:00
Michael Peter Christen
81f9b34da7 increaesed ability ot search for all images on a single server within
the p2p remote search
2014-09-15 20:33:22 +02:00
Michael Peter Christen
2c26013c50 better contentdom abstraction 2014-09-15 14:00:41 +02:00
Michael Peter Christen
6a8fb8190b changed default value for maximum number of connections to 50 2014-09-15 13:50:40 +02:00
Michael Peter Christen
ca8b2bf099 removed www and welcome servlet, these had been demo servlets and are
not needed any more
2014-09-15 12:48:58 +02:00
reger
03a7a29db3 limit OAI import urn resolver try for Deutsche National Library
The resolver service of National Library uses name space nbn, limit use of nbn-resolving.de accordingly to urn:nbn:
- add resolver for rfc's
2014-09-14 01:38:27 +02:00