Commit Graph

13600 Commits

Author SHA1 Message Date
luccioman
bfe753acea
Merge pull request #144 from him2him2/_fic_HTTPS
Update HTTP -> HTTPS in README.md
2017-12-02 08:45:42 +01:00
luccioman
c9d80b5b77 Prefer fine URL match over approximate URL mask regex on final filtering
Also prevent adding a redundant and CPU costly Solr url mask filter
query when possible
2017-12-01 11:52:52 +01:00
luccioman
0a120787e3 Improved accuracy of URLs search filters : protocol, tld, host, file ext 2017-12-01 11:19:31 +01:00
luccioman
d1c7dfd852 Fixed URL parsing with fragment and empty path 2017-12-01 09:48:42 +01:00
luccioman
e07ef1b610 Apply tld query modifier on Solr host_s mandatory field.
The filter has thus much more chances to be effective than when applied
on the optional field host_dnc_s.
2017-12-01 08:46:46 +01:00
luccioman
478e92deff Fixed url mask filter generated when protocol modifier is not null 2017-11-30 20:21:45 +01:00
luccioman
29de4a65d7 Refactored url mask filter build from query modifiers
For better readability and easier unit testing.
2017-11-30 09:20:32 +01:00
reger
a1879115dc upd to Jsoup-1.11.2 2017-11-26 22:01:42 +01:00
reger
d5a75537e4 remove redundant setting of timeout for remoteinstance
and replace depreciated updatesolrclient instantiation with recommended builder
2017-11-26 02:53:51 +01:00
luccioman
f01aac31fd Made possible to use https for remote search on peers with SSL enabled.
Default is still http to prevent any regressions, but a new setting is
available to choose https as the preferred protocol to perform remote
searches. 
New configuration setting 'remotesearch.https.preferred' is manually
editable in yacy.conf file or in Advanced Properties page
(/ConfigProperties_p.html).
Should be enabled as default in the future for improved privacy. 
Https could also eventually be used for other peers communications.
2017-11-24 14:10:41 +01:00
Ronald Eddy Jr
97dff48abf Update HTTP -> HTTPS in README.md
URLs were updated to use HTTPS protocol in README.md.
2017-11-23 00:54:36 -08:00
luccioman
01dca12d05 Upgraded apache POI dependency from 3.16 to 3.17 2017-11-22 09:07:36 +01:00
luccioman
e2f6427a63 Added a basic JUnit test for the Visio parser (vsdParser) 2017-11-22 09:06:16 +01:00
luccioman
1e9cdaabd4 Do locale neutral case conversion of HTML charset name.
Required to properly run on systems with default locale set to Turkish
language, as with this locale the 'i' character has different upper and
lower case flavors than with other locales.
2017-11-20 18:52:45 +01:00
luccioman
d41ad7af6f Restore initial locale at the end of a JUnit test case which modify it. 2017-11-20 18:50:49 +01:00
luccioman
7206f1ed71 Do locale neutral case conversions on domain names.
Required to properly run on systems with default locale set to Turkish
language, as with this locale the 'i' character has different upper and
lower case flavors than with other locales.
2017-11-20 18:47:46 +01:00
luccioman
398c66f06c Do locale neutral case conversions in MultiProtocolURL
For any relevant URL parts : host name, URL scheme, session ids or
technical parts (see https://url.spec.whatwg.org/#url-writing and
https://tools.ietf.org/html/rfc3986 for current standard references).

Remaining locale sensitive conversion used for detection of URL word
components in urlComps() makes sense but using detected language would
be preferable than using the default system locale.
2017-11-20 15:23:33 +01:00
luccioman
9531b83598 Do locale neutral case conversions in Classification
Required for people using Turkish language as their default system
locale, as with this locale the 'i' character has different upper and
lower case flavors than with other locales.
2017-11-20 09:48:46 +01:00
luccioman
bab5f0485f Added signing key to developer releases location. 2017-11-17 11:09:55 +01:00
luccioman
d22fc0d0a2 Updated lists of known sponsored and country-code TLDs.
Using current IANA reference list at
https://www.iana.org/domains/root/db .

As for previous update on known generic TLDs list, the generated URL
hashes on these domains stay the same but it improves performance of URL
hash computation for URLs on these domains.
2017-11-16 09:50:55 +01:00
luccioman
ac209cac2e Updated the generic top-level known domains list.
Using current IANA reference list at
https://www.iana.org/domains/root/db

The generated URL hashes on these domains stay the same but performance
is greatly improved as a DNS resolve request is required on URL hash
computation when the TLD part of the host name is unknown.

Hash computation mean time measured on 1541 sample URLs (one on each
TLD) and a computer with a DSL connection : about 230ms before change,
then only 20ms.
2017-11-14 09:42:09 +01:00
luccioman
938d8a9731 Added some JavaDoc 2017-11-14 09:24:13 +01:00
luccioman
c32ac9c4c7 Updated log path in informative message of stop script.
As highlighted by @Lew-Rockwell-Fan in issue #140, the two log paths
mentioned by the stopYACY.sh script were inconsistent.
2017-11-14 09:17:43 +01:00
luccioman
8f07df5f85 Upgraded com.twelvemonkeys.imageio dependencies from 3.3.1 to 3.3.2 2017-11-09 09:30:20 +01:00
luccioman
fcd57e2d0f Improved some JUnit tests isolation and resources release
The modified tests were successfull when run manually from an IDE such
as Eclipse, but failed occasionnally when run with maven as part of the
overall test suite.
2017-11-08 09:33:30 +01:00
luccioman
e0eda84c24 Remove old hard-coded holiday dates from DateDection class.
Replaced with rules based relative to current year as already done for a
part of the supported dates.
2017-11-07 19:02:09 +01:00
luccioman
f61260c4c7 Upgraded icu4j dependency from 59_1 to 60.1 2017-11-06 09:37:44 +01:00
luccioman
73977ec0fe Added a html parser charset detection unit test 2017-11-06 09:14:03 +01:00
reger
d14c47d4d3 upd to pdfbox-2.0.8.jar 2017-11-05 00:52:14 +01:00
luccioman
cb10daba92 Renamed Chinese & Greek lng files using ISO639-1 codes.
Previously named with their ISO 3166-1 country code : this way, when
setting language to "Browser" in ConfigBasic.html, it didn't work
properly when browser preferred language was Chinese or Greek as their
respective language codes are "zh" and "el" (not "cn" and "gr" which are
their country codes)
2017-11-04 11:06:05 +01:00
luccioman
4b61edff32 Added a help link to ISO 639-1 language codes list ref 2017-11-03 10:34:36 +01:00
luccioman
a994d439af Added description of spatial restrictions in search options 2017-11-02 08:57:00 +01:00
luccioman
46f37e38dc Customized Threads with generic name for easier monitoring. 2017-10-31 08:53:17 +01:00
luccioman
8a48f80909 Added language HTML attribute to the search home page. 2017-10-31 08:19:04 +01:00
luccioman
5ff76fdcb9 Fixed spelling 2017-10-31 07:52:30 +01:00
luccioman
2c3f0ff9e8 Updated search page keyboard shortcuts descriptions. 2017-10-31 07:44:37 +01:00
luccioman
af825e9ffc Use accessible labels for search home page radio buttons. 2017-10-30 08:07:59 +01:00
luccioman
046be566e1 Updated a license header typo. 2017-10-30 07:38:47 +01:00
Apply55gx
3c905a2a5c fix typo 2017-10-27 14:00:30 +02:00
luccioman
8e732d437c Enable HTTP Digest authentication for non admin users.
Also ensure authentication is not lost by Digest timeout when navigating
between index.html and search results page.

This way, running searches with extended features on a remote peer or a
password protected peer works with a regular user (with "Extended
search" rights). 
When authenticating on the search page with a user without "Extended
search" rights, it appears as authenticated, but has just its usual
access to the public search features.
2017-10-26 07:51:18 +02:00
luccioman
5161451a35 Stay authenticated when going to the search start page.
Otherwise, when authenticated as admin and navigating from search
results or admin pages to the search start page (/index.html), if
nothing is done on that page within HTTP Digest Auth timeout (about
2mn), then search is performed without authentication and so without
extended search features.
2017-10-24 09:54:54 +02:00
luccioman
d0bed78d02 Use the same top nav bar on index.html and search results.
Thus eventually including the same optional login link/status in the
search start page than in the results page, for the same convenient
login without the need to use the Administration section.
2017-10-24 09:34:03 +02:00
luccioman
d8eaf621cc Fixed blacklist returned location URL on empty parameters 2017-10-24 09:30:21 +02:00
luccioman
f678394ce5 Fixed loss of index page form values on 'more options' link click.
Restores the behavior introduced eleven years ago (see commit
479861a3cf) and lost by mistake 3 years
ago (see commit 617dd9c97b), when the
click handler started referencing a missing HTML id.
2017-10-23 18:28:11 +02:00
reger
b98acb33c3 upd to Solr 6.6.2 2017-10-22 20:00:00 +02:00
luccioman
af198b990b Added an optional login link/status to the search public top nav bar.
Thus allowing a more convenient way (wihout the need to go to the admin
section) to login when searching on your remote or password protected
peer and benefit from extended search features such as Heuristics,
Bookmarking or JavasScript resorting.

Can be disabled using the ConfigSearchPage_p.html.
2017-10-21 10:57:36 +02:00
luccioman
1de86cf1bf Fixed JPEG snapshot resizing when running on OpenJDK.
Resizing JPEG snapshot images through /api/snapshot.jpg failed when
running on OpenJDK, but rendered successfully with a Oracle JDK.
Details in mantis 772 ( http://mantis.tokeek.de/view.php?id=772 ).

Removing any alpha component (useless in snapshot images) from the
rendered resized image solves the issue.
2017-10-19 09:27:52 +02:00
luccioman
c1c4174816 Updated Java version information on Readme 2017-10-18 14:17:06 +02:00
luccioman
a17a418e78 Fixed NullPointerException cases on snapshot images parsing. 2017-10-18 08:31:18 +02:00
luccioman
285f0d6a39 Consistently encode snapshot image with format requested on the API.
Previously, calling /api/snapshot.png rendered JPEG encoded images.
2017-10-18 07:53:07 +02:00