Commit Graph

1203 Commits

Author SHA1 Message Date
JeremyRand
433217b33e Properly support multiple Boost Queries. (Previous code was broken because it concatenated multiple Boost Queries together rather than passing Solr an array.) 2016-05-20 20:17:51 -05:00
reger
d0a571bed2 del cytag trail for own index.html (save resource not used by default) 2016-05-19 01:59:00 +02:00
reger
7097dcbdbd cleanup hack for partial Solr update on multivalued datefields
has been fixed in Solr http://issues.apache.org/jira/browse/SOLR-8050
2016-05-06 02:47:04 +02:00
reger
f10ea3c155 clean-out unused SwitchboardConstants 2016-05-05 00:55:22 +02:00
reger
ef24593347 delete obsolete SEARCHRESULT busythread constants
not used since 29.05.2013 18:27:27
0c1a018bbd
2016-05-04 01:30:10 +02:00
reger
6ecc180299 fix rwi doubledom return best (highest) ranking 2016-04-12 03:55:43 +02:00
reger
d9adc2c255 load handler for Transparent Proxy on startup only if feature is activated
to save the resources and keep handler chain small if the feature is not used.
+add a warning message on settingsack_p page to restart on first activation
2016-03-25 05:26:48 +01:00
Michael Peter Christen
b89465d952 0N - basic dump upload servlet infrastructure, to share index dumps
within an experimental new sharing model
2016-03-11 18:12:13 +01:00
Michael Peter Christen
849ab671a9 0n: modified the p2p bootstraping process - rules had been too tight and
did not support the re-start of a network with just one principal peer.
2016-03-11 08:54:42 +01:00
Michael Peter Christen
a6bf0b1649 0N - added option to generate index export files for a specific number
of minutes in the past and reverted latest change. The export file dump
will now contain four data elements: f - first date of index entry write
date, l - last date of index write date, n - now-date of index dump
time, c - count of numbers inside the dump. '0N' denotes a series of
changes which will lead to the opportunity to exchange index data dumps
in a way that is needed to integrate ZeroNet index data. This will be
based on index dump sharing; that causes this commit.
2016-02-23 18:56:20 +01:00
reger
06d0e2aeb9 result heuristic (also used in greedy learning mode) to use outbound links if result is full index doc. Otherwise use default loader methode.
- Above brought up that parser start url parameter, declared as AnchorURL uses only methodes of parent object DigestURL (changed parameter declaration accordingly).
2016-02-16 02:05:58 +01:00
reger
caf9e98f09 put metadata dc_publisher in corresponding schema field 2016-02-14 21:13:25 +01:00
reger
6f0b073bf3 override detected language (statistic langdetect) only with TLD determided
language if langdetect probability is not high.
+ additionally truncate zh-cn / zh-tw returned by langdetect to 2 char ISO639-1 zh
used by YaCy
2016-02-07 21:16:22 +01:00
reger
535d4bf75f respect hidden attribute for file and smb directory listing
(hidden directories are not listed, effects crawling of local file system)
2016-02-04 19:16:00 +01:00
reger
a6617ad887 expand initRemoteCrawler() to terminate worker threads if called to deactivate
remote crawl.
On startup we save the resources for remote crawler if disabled. Once started
threads are running idle after disable remote crawl. Now threads are terminated
to save the resources also while disabeling during runtime.
+ remove empty class Channels
2016-01-28 23:14:09 +01:00
reger
ed3e16e092 apply remote result count config value to Bookmark Autosearch
+ prepare to make the widely unused Bookmark feature optional
2016-01-15 02:10:10 +01:00
Ryszard Goń
a98c395023 Add the Autocrawl thread 2016-01-14 00:50:23 +01:00
Ryszard Goń
1728cd30c6 Create autocrawl profiles 2016-01-12 16:28:34 +01:00
luc
571bc55937 Refactoring : use StandardCharsets constants instead of hard-coded
charset names.
2016-01-05 23:37:05 +01:00
reger
1af0e9ef74 remove workaround for Solr bug regarding multivalued date fields
fixed in 5.4.0
http://issues.apache.org/jira/browse/SOLR-8050
2016-01-03 01:11:27 +01:00
reger
a58d34a4e8 check error URL cache before adding errorDoc to index
- del obsolete related switchboardconstant
2016-01-02 05:03:57 +01:00
reger
cd26717ba2 fix low memory status hint (dht-in disabled)
http://mantis.tokeek.de/view.php?id=619
2015-12-29 20:38:45 +01:00
sixcooler
dce1cb65c4 Merge remote-tracking branch 'choose_remote_name/master' 2015-12-28 23:20:42 +01:00
reger
6d54eb3d36 skip loading document on crawl start for YMark bookmarks
by adding a constructor giving the already loaded document as parameter.
2015-12-26 01:15:07 +01:00
reger
45b9bd8403 adjust MultiProtocolURL.protocol detection to handle mailto with "://" in parameters,
and feeding hyperlinks to webgraph processing.
2015-12-21 04:42:26 +01:00
reger
dec3e6ad96 fix: adjust urlstub for mailto links
(skip protocol)
2015-12-19 20:13:33 +01:00
luc
8c4ab9c76b Added an option to eventually limit size of remote solr documents put to
local index. See mantis #626.
2015-12-16 02:20:03 +01:00
reger
28b8bc290a fix use of NETWORK_SEARCHVERIFY for rwi verification
was not used to set the searchevent parameter (done in SearchEventCache.getEvent)
- remove unused corresponding QueryParams.filterfailurls param.
2015-12-13 20:01:49 +01:00
reger
020630efd8 remove unused network scanner parameter from queryparameter
Search event is not using networkscanner 
(removed filterscannerfail param always init to false)
2015-12-13 02:50:08 +01:00
luc
ad5586f8f6 Merge branch 'master' of https://github.com/yacy/yacy_search_server 2015-12-08 03:35:36 +01:00
luc
8ebefa4233 Fixed MediaWiki import : DCEntry conversion to SolrInputDocument was
failing. Looks like it was broken since Commit
b43811d38c
2015-12-08 03:34:03 +01:00
reger
cdb8f3b10d make current ranking score value avail. to search interface / api
Update the result score result field with the result queue ranking value to reflect
the actual calculated/used score,
for rwi & solr stack results.
(calc. etc. is unchanged, it's just that result entry carries the latest val
as api retrieves the number from it)
2015-12-08 03:17:32 +01:00
Michael Peter Christen
ef8cd80593 fix for npe 2015-12-03 00:33:13 +01:00
reger
0347bfa71f Apply collection query constraint/modifiert to rwi result stack.
Collection is not available in pure rwi entries (but in local solr metadata)
But if user wishes to filter by query constraint also rwi shall adhere to this
(even if only rwi entries with parsed or solr received metadata may fit)
2015-12-02 22:57:59 +01:00
reger
ca3d26a401 harmonize wordsintitle & CollectionSchema.title_words_val calculation,
remove obsolete partial init of wordreference from urimetadata
2015-11-15 06:06:37 +01:00
reger
52a9040ae6 Sort out double keywords (dc_subject) early in parsed documents
- by direct using Set vs. List
- remove not neede String[] getter
2015-11-13 01:48:28 +01:00
sixcooler
646afe9183 do not store subfield *_coordinate + make all num-fields being docvalues 2015-11-10 20:45:33 +01:00
sixcooler
194df613de not using 'location' as defaultfacetfield - since we removed it being
default.
2015-11-10 20:43:58 +01:00
sixcooler
4a905ec134 fix to not let the AccessTracker-Log grow to much, but have enough data
to monitor.
(+gitignore-correction)
2015-11-10 20:27:17 +01:00
reger
a60b1fb6c2 differentiate api call getLocalPort() from getConfigInt() 2015-10-31 23:09:03 +01:00
reger
11f3666660 increase use of pre.defined CATCHALL_QUERY string 2015-10-31 19:44:31 +01:00
reger
a58ee49307 Optimize internal imagequery focus on using content_type to select images
(in favor of url file extension)
2015-10-31 19:18:46 +01:00
Michael Peter Christen
151ccd50a9 fix for image size field values (must be multi-valued) 2015-10-14 15:16:16 +02:00
reger
43c27aa550 upd to solr/lucene 5.3.1 2015-10-03 23:20:33 +02:00
Michael Peter Christen
3d7dd9d3aa follow-up to latest commit: also flush the search cache if all crawls
had been terminated.
2015-10-01 13:21:28 +02:00
Michael Peter Christen
c737ff235d in case that the include_string contains several entries including
1-char tokens and also more-than-1-char tokens, then remove the 1-char
tokens to prevent that we are to strict. This will make it possible to
be a bit more fuzzy in the search where it is appropriate.
2015-10-01 13:09:33 +02:00
reger
7889fc2389 Hack to prevent Solr issue on partial update on a document containing multivalued date field
(regardless if these fields part of update).
Switch partial update option off in postprocessing if schema contains *_dts (multivalued date field).
see http://mantis.tokeek.de/view.php?id=601
2015-09-13 20:23:15 +02:00
reger
3428b6f13b improve filtering by filetype navigator.
The used url-filter for filetype doesn't require ".ext" resulting in too many matches,
add a sort-out filter for RWI results.
2015-09-07 02:36:22 +02:00
reger
e37a4f0b3d prevent metadata records in index w/o valid url
by throwing MalformedURL exception on URIMetadataNode creation
2015-09-06 22:19:05 +02:00
reger
802ccaead6 fix init of error cache, use latest faildates => load_date_dt 2015-09-02 02:36:31 +02:00