Michael Peter Christen
d0abb0cedb
enabling all crawl profiles in all network modes
...
also: increased default internet crawl speed to
4 urls/s/host
2020-12-19 01:00:51 +01:00
Michael Peter Christen
baad56d83d
beautified default peer names
2020-12-14 02:08:49 +01:00
Michael Peter Christen
43a9f4f574
updated solr 6.6.6 -> 7.7.3
...
dropped GSA support (GSA API is still in YaCy Grid)
The 6.6.6 solr index works without migration also with 7.7.3
2020-12-12 02:06:43 +01:00
Michael Peter Christen
c0d9a3e9a7
turned HostBrowser into a admin-only page, now called IndexBrowser
...
This was required because spiders and bots crawled through this page and
created load on the peer without use for the user or the YaCy network.
2020-12-11 00:50:52 +01:00
Michael Peter Christen
d359d521a1
fixed warc importer
...
The importer tried to import a gziped files as plain warc.
It will now check the file extension and use a unzip automatically
on-the-fly.
2020-12-10 11:19:25 +01:00
Michael Peter Christen
e54ab39958
Going back to basic authentication for console/shell commands
...
This does not affect security because:
- it is going to localhost only
- only users who have already access to the pw hash can do this
- no clear text pw is transmitted because that is not stored anywhere
The switch to basic is required because these commands are required
in the context of hosting on root servers and docker containers
where a password change must be done. But the password shell command
was not working without password which made the concept unusable.
This deficit made it virtually impossible for root server operators
to use YaCy because they had been unable to set up a proper password.
2020-12-09 02:36:55 +01:00
Michael Peter Christen
6271e9122c
javadoc fix
2020-12-09 02:22:47 +01:00
Michael Peter Christen
e0f4e3fd9a
enhanced ability to debug the code
2020-12-09 02:22:30 +01:00
Michael Peter Christen
eea2d71851
prevent creation of auth schema factories every time a servlet is called
2020-12-06 01:49:34 +01:00
Michael Peter Christen
fcc9386ed3
enhanced the (already fast!) png exporter
2020-12-03 12:18:07 +01:00
Michael Peter Christen
4e9b425f98
missing fix for latest commit
2020-12-03 00:40:51 +01:00
Michael Peter Christen
3213d9db37
updated jetty from 9.4.17 to 9.4.35
...
and fixed a bug in ServerSideIncludes that appeared only in that recent
version of jetty
2020-12-03 00:21:15 +01:00
Michael Peter Christen
787fec0658
reduced complexity - removed concurrency in sort
2020-12-02 18:39:45 +01:00
Michael Peter Christen
cef5fde343
adding message to UI to make port change transparent
2020-12-02 18:05:38 +01:00
Michael Peter Christen
52228cb6be
added a gc to cleanup process (once every 10 minutes)
2020-12-02 00:13:00 +01:00
Michael Peter Christen
22841ffbf1
creating a threaddump during every cleanup process
...
to be able to find out what a peer did (not) last time before a crash
2020-12-01 03:00:24 +01:00
Michael Peter Christen
36e616271b
do better documentation on how to set a default password
2020-12-01 02:18:08 +01:00
Michael Peter Christen
df2bf9ef28
try to fix maven build error
2020-11-29 14:24:33 +01:00
Michael Peter Christen
264bab6700
trying to fight the UI unavaiability
...
this path addresses a possible issue with too many open connections to
remote peers
2020-11-29 14:15:34 +01:00
Michael Peter Christen
7947baeb49
removed all remaining deprecation warnings
2020-11-23 00:03:18 +01:00
Michael Peter Christen
c0f6d6e11d
removed one deprecation warning for jetty library initializing ssl
...
server port
2020-11-22 23:27:58 +01:00
Michael Peter Christen
133440a7a6
some debug lines
2020-11-22 23:12:04 +01:00
sgaebel
3431f91db9
removes unused 'unused' tokens
2020-08-04 20:09:34 +02:00
sgaebel
fc03c4b4fe
removes some warning and unused objects
2020-08-03 20:44:31 +02:00
sgaebel
4a495df63a
removes some deprecation-warnings
2020-07-31 17:28:06 +02:00
sgaebel
dd9d4b1188
replace org.junit.Assert.assertThat by
...
org.hamcrest.MatcherAssert.assertThat from hamcrest 2.2 to avoid
deprecation-warning
2020-07-28 19:09:26 +02:00
sgaebel
df9ea0a42a
removes some warnings: unused imports, params
2020-07-27 22:20:49 +02:00
sgaebel
9bc2297161
fixes deleting during recrawl
2020-07-22 22:15:00 +02:00
sgaebel
80785b785e
adds deleting during recrawl
2020-07-09 19:32:16 +02:00
Michael Peter Christen
e0ad8ca9da
replaced json library from JSON.org with libandroid-json-java
...
This fixes https://github.com/yacy/yacy_search_server/issues/347
2020-04-24 11:45:25 +02:00
Michael Peter Christen
ea8df27e95
modified org.json.* library to fit into the YaCy environment
...
as drop-in replacement.
Also made some fixes and enhancements to the library.
2020-04-24 11:42:06 +02:00
Michael Peter Christen
60dc1241a3
added org.json.* library
...
from https://android.googlesource.com/platform/libcore/+/refs/heads/master/json/src/main/java/org/json
as a preparation step for
https://github.com/yacy/yacy_search_server/issues/347
2020-04-24 10:28:43 +02:00
Michael Peter Christen
053e54a2c7
grand CORS for json files
2019-11-05 11:50:56 +01:00
Michael Christen
cfa27d2fd5
fixed links
2019-10-20 20:20:50 +02:00
Michael Christen
cb20aa7e54
removed donation message in search result column
2019-10-17 01:35:44 +02:00
Michael Christen
25227676ae
removed some warnings
2019-09-28 02:07:08 +02:00
luccioman
6b45cd5799
New optional crawl filter on the URL a doc must match to crawl its links
...
For finer control over which parsed documents can trigger an addition of
their links to the crawl stack, complementary to the existing crawl
depth parameter.
2019-05-01 08:54:19 +02:00
luccioman
d16bc99835
Added "Show Metadata" links to the ViewFile.html links mode
...
To conveniently follow parsed links in the file viewer
2019-04-18 15:31:38 +02:00
luccioman
a5771b1f14
Made SNI extension user configurable without the need for server restart
...
TLS Server Name Indication (SNI) extension activation can now be
configured with the new Settings_p.html?page=httpClient administration
page.
SNI extension is also now enabled by default, as in 2019 the
unrecognized_name(112) alert is more properly handled by major web
servers TLS implementations, following the RFC 6066 standard.
Related YaCy issues : #153 #189 and #272
JDK 1.7 bug :
https://bugs.java.com/bugdatabase/view_bug.do?bug_id=7127374
Apache httpd issue :
https://bz.apache.org/bugzilla/show_bug.cgi?id=56241
RFC 6066 : https://tools.ietf.org/html/rfc6066#section-3
2019-04-14 15:41:13 +02:00
luccioman
e90405b6f0
Support parsing audio URLs without file extension
...
Added also a Junit for the audio tag parser
2019-04-09 11:40:21 +02:00
luccioman
a8316c79da
Allow JS resorting of search results by unauthenticated users
...
Acces rate limitations to this search mode by unauthenticated users are
set low by default to prevent unwanted server overload but can be
customized through the SearchAccessRate_p.html configuration page
Fixes #291
2019-04-03 14:21:53 +02:00
luccioman
0ab2b49c31
Made /yacysearch access rate limitations user configurable
...
With a new admin page at /SearchAccessRate_p.html in menu Network Access
> Local Search > Access Rate Limitations
2019-04-02 17:42:50 +02:00
luccioman
5b7e41202a
Added Solr GSA writer support for responses from remote instances
2019-03-27 18:23:41 +01:00
luccioman
4d8a948455
Properly close PDF snapshots loaded with pdfbox library
2019-03-22 09:50:30 +01:00
luccioman
74e6d6e984
Added Solr GrepHTML writer support for responses from remote instances
2019-03-20 18:24:16 +01:00
luccioman
5e6501974d
Added Solr snapshots writer support for responses from remote instances
2019-03-19 11:25:44 +01:00
luccioman
384c37102c
Improve accuracy of total results count on latest pages in Stealth mode
...
Previously, when mixing results from local RWI and local Solr (Stealth
mode), total local Solr count could be ignored on last result pages,
when the page offset was higher than local Solr count but lower than
total RWI count.
2019-03-04 10:05:47 +01:00
luccioman
5e9a08355a
Improved logging for federated search
...
- Do not use spaces in logger identifier name so the log level can be
configured in yacy.logging
- Hold the logger instance to avoid the logging system to look for it
from its name at each appended log message
2019-02-02 09:59:24 +01:00
luccioman
9782a98a9c
Added the possibility to customize facets sort type and direction
...
Previously search navigators/facets elements were sorted only by counts.
Now from the ConfigSearchPage_p.html admin page, sort direction
(ascending/descending) and type (on counts or labels) can be customized
independently for each navigator.
2019-01-24 18:43:06 +01:00
sgaebel
c2398fd890
remove warnings: 'Statement unnecessarily nested within else clause'
2019-01-10 20:02:57 +01:00