Commit Graph

8625 Commits

Author SHA1 Message Date
luccioman
e357ade47d Reduced memory footprint of text snippet extraction
By not parsing and storing at first all sentences of a document, but
only on the fly the ones necessary to compute the snippet.
2018-05-13 10:29:52 +02:00
luccioman
e115e57cc7 Reduced text snippet extraction processing time.
By not generating MD5 hashes on all words of indexed texts, processing
time is reduced by 30 to 50% on indexed documents with more than 1Mbytes
of plain text.
2018-05-11 15:42:53 +02:00
sgaebel
4b79851e12 corrected icons_sizes_sxt to SolrType.string 2018-05-01 14:04:15 +02:00
luccioman
3b89c232db Easier tracking of longest text snippets initializations
When text snippets statistics are enabled and FINE log level is enabled
on the TextSnippetStatistics class.
2018-05-01 09:58:05 +02:00
luccioman
3c4344cb12 Fixed text snippet max init time statistic rendering 2018-05-01 09:39:41 +02:00
reger
a8234b7ea7 Make sure for image resource url enabled index image pixel size fields are filled
if at least one of the image size fields is enabled in index (images_height_val,
images_width_val, images_pixel_val). 
Previously all fields were required to be enabled (hint: default setting 
is height + width enabled)
2018-04-30 04:59:34 +02:00
luccioman
e67df103b5 Removed more remaining uses of deprecated Seed.getIP() function. 2018-04-29 08:26:53 +02:00
luccioman
addd18c993 Removed some remaining uses of deprecated Seed.getIP() 2018-04-26 09:39:30 +02:00
luccioman
c35d0568b6 Support for preferred https in peers communication on more operations 2018-04-24 08:08:24 +02:00
luccioman
e914d17aca Updated call to function deprecated since commons-codec version 1.11 2018-04-23 08:07:56 +02:00
luccioman
a3ec7a7a5f Added analysis optional setting to compute statistics on text snippets
Thus producing some basic stats on processing times for snippets
generation and counts on snippets per source type.
2018-04-15 09:55:08 +02:00
luccioman
1889d484de Added Solr HTML writer support for responses from remote instances 2018-04-12 09:23:00 +02:00
luccioman
2af3bf79c7 Improve rendering of remote Solr admin URLs
- properly handle IPv6 loopback address replacement
 - replace loopback address or host only when accessing peer remotely
 - replace loopback part with the peer hostname as requested rather than
with its seed public IP as this works better for Intranet mode and when
peer is behind a reverse proxy.
2018-04-10 11:15:31 +02:00
luccioman
bb74de7d59 Removed unnecessary "/admin" suffix from remote Solr instance admin URL
For quite quite a long time now, the Solr /admin URL suffix indeed
redirects to the Solr base context (see
https://issues.apache.org/jira/browse/SOLR-3337)
2018-04-09 00:01:45 +02:00
luccioman
0d34034f17 Ensure an embedded Solr is available for Solr dump/restore operations
Otherwise, these operations triggered NullPointerException when only an
external Solr index is attached.
2018-04-07 13:42:06 +02:00
luccioman
d92b191942 Ensure no remote Solr is attached before "Shut Down and Re-Start Solr"
Otherwise once this operation is applied, the remote Solr(s) instances
are deconnected and the embedded Solr is connected even if disabled by
setting "core.service.fulltext".

Also use constants for related default setting values.
2018-04-06 20:34:54 +02:00
luccioman
26d8ad591c Adjusted Solr select servlet output when using an external Solr only
- Use the EnhancedXMLResponseWriter only when requested output is "exml"
- Use the Standard Solr writers when possible, for example for json, xml
or javabin output formats
- Return an error when the requested format can not been rendered with
an external Solr server only

Important : this modification is necessary for peers using exclusively
an external Solr server to be reachable as robinson targets in p2p
search, as the binary format ("javabin") is the default Solr exchange
format for peers.
Before this, when a peer requested a remote one attached only to an
external Solr (no embedded one), it ended with "Invalid type" error, as
the remote peer answered with xml although binary format was requested.
2018-04-06 15:16:54 +02:00
luccioman
69690c13a0 Optionally allow external Solr server with self-signed certificate
This is necessary when you want to attach to a dedicated external Solr
server protected with basic http authentication and requested over https
but having only a self-signed certificate.
2018-04-04 18:16:26 +02:00
luccioman
b882f85900 Fixed NPE case in Solr select servlet on external Solr only setup
Regression introduced with commit
0d7625ecfb
2018-04-03 15:36:17 +02:00
luccioman
2fd4d05e2f Added a shared Java constant for setting key server.servlets.called 2018-04-02 15:16:10 +02:00
luccioman
ba9cd14516 Removed hard-coded patch for Solr 5.0 on ranking boost function
The current default boost function
(`recip(ms(NOW,last_modified),3.16e-11,1,1)`) for the Date ranking
profile is indeed working fine.
What can trigger the error `unexpected docvalues type NUMERIC for field
'last_modified'` is the previous default boost function (quite old now)
or any custom one using the Solr `ord` or `rord` functions on the
last_modified field.
Then the problem was that the migration code in the Switchboard supposed
to detect the old date boost function was incorrect (one trailing right
parenthesis in excess), so the deprecated function remained.

This fixes issue #169.
2018-03-26 16:24:27 +02:00
luccioman
fb3032c530 Added a crawl filtering possibility on documents Media Type (MIME) 2018-03-23 10:28:19 +01:00
luccioman
e45afedee4 Added support for enclosures (media links) to the RSS loader 2018-03-21 08:22:29 +01:00
luccioman
aaefd5219c Reduce log verbosity of RSS loader on feed items with no link 2018-03-20 10:09:17 +01:00
luccioman
cf62b571bd Added RSS reader support for enclosure feed item sub element.
Enclosure element (see
http://www.rssboard.org/rss-specification#ltenclosuregtSubelementOfLtitemgt
) can be seen for example in podcasts feeds.
2018-03-20 07:38:29 +01:00
luccioman
e5f5de0fc7 Added some JavaDoc to the RSSMessage class. 2018-03-19 11:15:31 +01:00
luccioman
0d7625ecfb Handle Solr fields restrict and alias in YaCy html and exml writers
Thus allowing for example to read more easily the local Solr index full
metadata in HTML by restricting if desired to some fields of interest.

See Solr documentation about the 'fl' (Field List) parameter at
https://lucene.apache.org/solr/guide/6_6/common-query-parameters.html#CommonQueryParameters-Thefl_FieldList_Parameter
2018-03-16 11:35:42 +01:00
luccioman
3da2739bbd Parse and index more common audio metadata text tag fields. 2018-03-15 09:59:57 +01:00
luccioman
846aba00fa Added parsing of URLs eventually present in audio metadata tags 2018-03-13 23:08:52 +01:00
Michael Peter Christen
187075b878 added nav filter 2018-03-10 15:46:53 +01:00
luccioman
bcbd0ae1a4 Enabled partial parsing of audio resources. 2018-03-01 20:50:44 +01:00
luccioman
fda0189613 Updated audio file extensions with ones recently added to audioTagParser 2018-02-28 13:46:40 +01:00
luccioman
978e2be95b Let a chance for other parsers on audioTagParser error
As done in all other parsers, eventually falling back in the end to the
genericParser which creates a minimal index entry.
2018-02-28 12:27:17 +01:00
luccioman
9e5846a26e Small fix on svg parser error message 2018-02-28 12:23:52 +01:00
luccioman
11611dbdcf Reuse existing File copy function to handle audio parser tmp files 2018-02-28 11:58:32 +01:00
luccioman
f77f8f40f9 Factored audio parser tag processing 2018-02-28 08:19:13 +01:00
luccioman
9a7a353d0e Removed some unnecessary intermediate list creation on array copy. 2018-02-28 07:49:40 +01:00
luccioman
fb6457f5bc Fixed NPE case when on audio resource parsed with null tag 2018-02-28 07:31:32 +01:00
luccioman
c3ff50c17a Updated the list of audio file formats supported by the audioTagParser
Follows upgrade to Jaudiotagger dependency to version 2.2.5.
2018-02-27 18:04:12 +01:00
luccioman
1b90479a76 Added missing vocabulary navigator increment on results from RWI 2018-02-23 11:36:03 +01:00
luccioman
46c9da6428 Allow creation of vocabularies from remote CSV file URLs. 2018-02-21 08:41:13 +01:00
luccioman
17c7a85f18 Make StreamResponse usable in Java try-with-resources statements 2018-02-21 08:38:35 +01:00
luccioman
b67742336e Provide user interface messages on vocabulary creation read/write errors 2018-02-19 11:48:40 +01:00
luccioman
3e8dd90211 Use https rather than http in links and queries to openstreetmap.org 2018-02-15 19:14:07 +01:00
luccioman
3a973dbb23 Removed unused import 2018-02-14 09:27:17 +01:00
luccioman
e9527cd0e5 Reuse the same Pattern instance when matching multiple key/values 2018-02-14 07:14:25 +01:00
luccioman
dbf4c1cd76 Improved blacklist entries editing operations :
- Fixes issue #160 : handle properly syntax exceptions with a user
friendly message
- Fixes loss of information on multiple blacklist entries editions
- Fixes loss of entries when moving entries from one list to another
2018-02-13 18:24:26 +01:00
reger
87077b8fb6 Adjust and move Language Navigator to be member of the navigatior plugin
list.
2018-02-12 00:16:34 +01:00
luccioman
eb20589e29 Fixed issue #158 : completed div CSS class ignore in crawl 2018-02-10 11:56:28 +01:00
luccioman
0cdee4e26a Fixed loss of "meanCount" search param when using facets or page buttons
Then on new search queries, no suggestions at all could be displayed.
2018-02-08 08:07:30 +01:00