Commit Graph

3690 Commits

Author SHA1 Message Date
reger
6f8c3ccea4 improve url hash computation for file path with mixed java & windows
file.separator to compute equal hashes (by normalizing path for computation)
+ expand test case for to check mixed java / windows file url notation
like e.g. file:///c:/test/file.html vs. file:///c:\test/file.html
- relates partially to http://mantis.tokeek.de/view.php?id=692
2016-09-25 22:08:12 +02:00
reger
efcb6a1e74 fix supported mime XML -> xml for rssParser (mime normalized to lower case for comparison)
+ add mime text/xml as in use for rss in the wild
2016-09-23 23:37:12 +02:00
luccioman
b3b75b0498 Accessibility : add a customizable alternative text to YaCy log
Applied W3C recommendations :
https://www.w3.org/TR/html51/semantics-embedded-content.html#a-link-or-button-containing-nothing-but-an-image
and
https://www.w3.org/TR/html51/semantics-embedded-content.html#logos-insignia-flags-or-emblems
2016-09-22 16:08:33 +02:00
luccioman
f2bc1b268d Updated URL fragment validation rules according to current standards
See RFC 3986 (https://tools.ietf.org/html/rfc3986) or URL living
standard (https://url.spec.whatwg.org/)
2016-09-22 11:28:33 +02:00
luccioman
b1b8e69da8 Fixed NullPointerException cases 2016-09-22 11:25:33 +02:00
luccioman
3ee4f56c39 Improved ErrorCache behavior when switching networks
Even after network switch, ErroCache was still holding a reference to
the previous Solr cores, thus becoming useless until next YaCy restart.

Initial error cache filling with recent errors from the index was also
missing after the swtich.
2016-09-22 09:07:07 +02:00
luccioman
7d5ba2afa4 Added some JavaDoc and moved crawlStacker close at the right place. 2016-09-22 08:21:14 +02:00
luccioman
8edbcd8ad4 Log eventual Solr instances close errors.
We do not want to block on this kind of error, but this should not
silently fail as it may have later consequences.
2016-09-22 08:20:01 +02:00
reger
330768c8a2 fix for solr write.lock after mode change http://mantis.tokeek.de/view.php?id=686
The embedded core holds a lock on the index and must be closed. Earlier commit
comment states that core should be closed with solr instance instead on close 
of connector.
Adjusted the InstanceMirror.close() to take care of closing the embedded 
instance to release the lock.
In 2 routines of fulltext this was already explicite implemented (disconnectLocalSolr).
Now this disconnect is part of the InstanceMirror.close().
2016-09-22 00:16:22 +02:00
reger
585d2a6441 test case: for NewsPool to check the id modificator (for unique id)
and observe the distribution order .. hands on.
+ add test/DATA to gitignor
2016-09-20 01:55:56 +02:00
luccioman
de5c873e38 Removed unused JavaScript file docs.min.js
This file is used by Bootstrap documentation website
(http://getbootstrap.com/) but is not part of the Bootstrap distribution
and has not be included in a Bootstrap based application.
2016-09-20 00:17:42 +02:00
Michael Peter Christen
df51e4ef07 Merge branch 'master' of git@github.com:yacy/yacy_search_server.git 2016-09-19 11:01:58 +02:00
Michael Peter Christen
e063aaf97f enable fuzzy search, solr style (append a ~ to get a fuzzyness on the
word)
2016-09-19 11:01:39 +02:00
reger
ff6589fc0f test case: simulating multi word query for local rwi index
Purpose of the test case is to be able to (controlled) analyse the rwi ranking for
multi word searches (with focus on posintext and word-distance ranking)
2016-09-18 00:59:27 +02:00
reger
e990297d2e avoid NPE on hello message with missing "yourip" key
http://mantis.tokeek.de/view.php?id=684
2016-09-15 23:26:25 +02:00
reger
e51ab8c7aa hack to generate a unique message-id for messages created in the same second
by optionally add a 1 second offset counter to the current time (which is
used as the unique id part)
2016-09-15 02:59:32 +02:00
Michael Peter Christen
b82300358a removed version number check because it does not work any more if
version numbers are expressed in a different way as we expect. That
could cause that YaCy does not run on systems which are appropriate but
we simply do not understand the version string.
2016-09-14 16:32:57 +02:00
Michael Peter Christen
2107674999 Merge branch 'master' of https://github.com/yacy/yacy_search_server.git 2016-09-14 16:24:55 +02:00
Michael Peter Christen
0d28f563f4 fix for java version "9-ea" 2016-09-14 16:24:32 +02:00
reger
3b694b3935 add some javadoc to rwi wordreference distance, position
to remember facts for http://mantis.tokeek.de/view.php?id=683
Init missing word position to 0 like in other non text body words
2016-09-14 00:36:19 +02:00
reger
a4465c97d6 as requested, disable/remove old swf parser
http://forum.yacy-websuche.de/viewtopic.php?f=8&t=5861#p33098
2016-09-13 02:47:36 +02:00
reger
7f63fc50f3 prepare a IndexSegment test case for RWI index testing
+ prevent NPE in Segment.clear() on missing embedded solr instance.
2016-09-11 23:25:44 +02:00
reger
96467c5467 remove not needed counter in Tokeninzer (completing last changes)
including a small change, word posintext counting. 
We remember/store 1st posintext. Previously following words got a handle (posintext)
excluding found. Now it just counts and assigns true posintext as handle (posintext)
2016-09-10 18:23:09 +02:00
luccioman
d66b0f7b7b Fixed french messages encoding in YaCy tray.
Also added the missing french translations.
2016-09-09 07:43:33 +02:00
reger
7efb66ee10 adjust the WordReference.join wordsintext calc to take the max (instead of sum)
The reference is for the same url (add same for title and phrases).
+ del redundant join() procedure
2016-09-08 02:29:48 +02:00
luccioman
0a9ff14d96 Fixed NullPointerException case and added Javadoc 2016-09-07 10:03:48 +02:00
luccioman
06d4f93d03 Merged master into postprocessing branch 2016-09-07 09:28:37 +02:00
Michael Peter Christen
b73d2db914 Merge branch 'master' of https://github.com/yacy/yacy_search_server.git 2016-09-07 06:49:15 +02:00
Michael Peter Christen
25a3c7a6d0 catch exception and write end of object 2016-09-07 06:48:52 +02:00
reger
272cdd496a reactivate sentence counter in WordTokenizer for phrasepos ranking,
by counting punktuation (delivered as 1 char word) again.
2016-09-07 02:16:16 +02:00
Michael Peter Christen
5e165a8150 removed unused imports 2016-09-06 18:46:24 +02:00
Michael Peter Christen
c716648c78 enhanced json encoding of strings 2016-09-06 18:45:29 +02:00
Michael Peter Christen
6139bd85a8 fix for broken facet names 2016-09-06 17:19:54 +02:00
Michael Peter Christen
5060f9fee9 fix for too long snippets 2016-09-06 09:05:39 +02:00
Michael Peter Christen
8681cee3f3 fix for bad comma 2016-09-06 09:00:35 +02:00
Michael Peter Christen
db6d8fc197 fix for bad json 2016-09-06 07:44:38 +02:00
Michael Peter Christen
8f4a341735 Merge branch 'master' of https://github.com/yacy/yacy_search_server.git 2016-09-06 06:58:17 +02:00
Michael Peter Christen
9934f546bb added default fl to solr query, removed large texts retrieval and
changed snippet to description tag if no other description is available
2016-09-06 06:56:51 +02:00
reger
120bf7e6e2 implemented RWI WordReference to return the word position value (was always left empty)
This is needed and enables existing word position ranking for RWI.
The upcoming concurrency issue in word position min/max calculation were eliminated
by iterator.hasHext check before next() access.
2016-09-06 03:18:02 +02:00
reger
e310ec5f70 fix posInText ranking calculation to score 0 on no position info
+ fix Word posInText calc in Tokenizer to start with 1
+ test case
2016-09-06 00:05:59 +02:00
luccioman
74f9927ddc Merge remote-tracking branch 'origin/master' into dist_macOS 2016-09-05 22:42:17 +02:00
reger
51c077f493 adjust the getTopics() and getTopicNavigator() to current useage
- move the maxcount limit restriction completely to getTopicNavigator (as there not used in getTopics)
- let search servlet use getTopics by default (w/o RWI connected check, as of now, Topics are available w/o any additional index interaction)
2016-09-05 00:07:01 +02:00
reger
39dd244693 fix ConcurrentScoreMap.set() calculation of totalCount()
+ test case
2016-09-04 22:18:07 +02:00
reger
ebf818ad95 log a error on aborted news publish (due to duplicate news.id)
+ change printed err msg to log entry in PeerAction.processPeerArrival
2016-09-04 06:42:48 +02:00
reger
cc2d9dd3f1 reactivate the use of included-in-topwords boost in postRanking
+ changed the postRanking to add one score only if word appears more as one time.
+ getTopics() unused code block rem'd (save performace)-> routine needs rework !
2016-09-04 00:09:45 +02:00
luccioman
39ea28adfd Merged master to dist_macOS branch. 2016-09-03 15:22:57 +02:00
luccioman
8255e91c99 Fixed serverClassLoader.findClass method
htroot is a supposed to be a subfolder of appPath and not of dataPath,
as assumed in other places where htroot is loaded. This issue was not
visible when dataPath and appPath are equals.
2016-09-03 15:21:02 +02:00
reger
6801673a07 apply postranking media search boost only on media queries 2016-09-03 03:37:40 +02:00
luccioman
1dc4306058 Fixed indentation for better readability. 2016-09-02 11:23:02 +02:00
luccioman
8c49a755da Postprocessing refactoring
Added Javadocs to refactored methods.
Added log warnings instead of silently failing some errors.
Only fill collection1hosts when required ( shallComputeCR true).
2016-09-01 15:40:28 +02:00