Marown/yacy_search_server
1
0
Fork 0
mirror of https://github.com/yacy/yacy_search_server.git synced 2024-09-19 00:01:41 +02:00
Code Issues Actions Packages Projects Releases Wiki Activity
14,126 Commits 1 Branch 20 Tags 836 MiB
69adaa9f55
Commit Graph

4382 Commits

Author SHA1 Message Date
sgaebel
69adaa9f55 makes our HTTPClient closable 2021-10-31 23:06:02 +01:00
sgaebel
fc4275f901 handle all references for client, response, request to be able to close 
them
 2021-10-31 23:05:50 +01:00
sgaebel
e7d3a363f2 refactor to use finish() 2021-10-31 11:22:35 +01:00
sgaebel
4fc876f4a3 revert back to use EntityUtils.consumeQuietly - as it simply closes the 
underlying stream
 2021-10-31 11:22:28 +01:00
sgaebel
4f0392e93e refactor use of AuthSchemeProvider 2021-10-31 11:21:59 +01:00
sgaebel
b74f337859 removes double setting of UserAgent 2021-10-31 11:21:06 +01:00
sgaebel
965748fefb some refactoring using try with resources 2021-10-31 11:20:28 +01:00
Michael Peter Christen
552ab7051b fix for warc importer 2021-10-25 19:35:15 +02:00
Michael Peter Christen
3c86b7b780 attempt to make a Mac Release using gradle 
This is almost working with many workarounds:
- run rm lib/yacycore.jar
- run ./gradlew clean build bundleNative
- run ant clean all
- run again rm lib/yacycore.jar
- run ./fixMacBuild.sh

The build is then inside build/mac/YaCy.app

Right now this works so far but it does not have the correct release
number inside.

Target is to make this working for Windows releases and to embedd jre
entirely.
 2021-10-25 18:37:39 +02:00
Michael Peter Christen
999c819e3e Merge branch 'master' of https://github.com/yacy/yacy_search_server.git 2021-10-24 20:50:14 +02:00
Michael Peter Christen
fd770e90e2 spike to identify paths for YaCy within mac application bundles 2021-10-24 20:49:59 +02:00
Michael Peter Christen
d19872fd26 making sure that crawl queues are closed correctly to prevent data loss 2021-10-14 00:30:04 +02:00
sgaebel
90507c0fdc comments out printing query params to std.out 2021-10-04 18:03:06 +02:00
Michael Peter Christen
be0aebad84 fixes https://github.com/yacy/yacy_search_server/issues/424 2021-10-04 14:38:49 +02:00
Michael Peter Christen
63ad8ce6b2 removed ymarks 
had not been used since a long time
 2021-09-16 22:23:51 +02:00
Michael Peter Christen
ef5a71a592 enhanced crawl start response time 
for very very large crawl start lists
 2021-09-16 21:01:01 +02:00
Michael Peter Christen
4cadd557dc removed synchronization in table creation 
to avoid possible deadlocks when handling OnDemandOpenFileIndex
which happens quite often during wide crawling
 2021-09-15 19:34:49 +02:00
admin
9b7668fa58 reduced memory footprint during indexing/crawling 2021-08-24 12:24:52 +02:00
Michael Peter Christen
e6a87e0426 enhanced crawler 
a main problem when crawling is long waiting time cuased by crawl-delay
values from robots.txt entries. that attribute is not supported by
google and interpreted by yandex and bing in different ways. In large
crawls there is always one host which blocks the whole crawl with
extreme large values. YaCy now still obeys crawl-delay but limits them
to 10 seconds.
Additionally the blocking logic when loading new robots.txt was analyzed
and a deadlock was removed. Furthermore the construction of new queue
lists was redesigned and it was ensured that always a large list of
different hosts for host-balancing is provided for the loader.
 2021-08-17 15:23:21 +02:00
Michael Peter Christen
e9c5e78868 replaced new Number(Number) with Number.instanceOf 
to remove deprecation warnings for Java 9
 2021-08-08 00:39:03 +02:00
Michael Peter Christen
9e13d77de4 removed call to class.finalize() because of deprecation in java 9 
next: removal of finalize() implementation
after testing with assert false
 2021-08-07 18:57:49 +02:00
Michael Peter Christen
9ef4503672 fixed some newInstance() warnings 
.. by adding .getDeclaredConstructor()
 2021-08-07 18:46:53 +02:00
Michael Peter Christen
1d41380f0a better support for mac-specific tray functions in java 9 2021-07-12 17:27:59 +02:00
Michael Peter Christen
e81b770f79 enabled crawl starts with very large sets of start urls 
i.e. 10MB large url list with approx 0.5 million start points
 2021-06-30 10:45:58 +02:00
Michael Peter Christen
c623a3252e fix for jdk 14 bug 2021-04-23 09:11:03 +02:00
Michael Peter Christen
dbd211a1ad removed/replaced reflection in memory tool 2021-04-22 20:24:13 +02:00
Michael Peter Christen
1cdb21592b added hazelcast and some modifications to align legacy YaCy with 
YaCyGrid
 2021-04-15 20:39:22 +02:00
Michael Christen
42ea2a1c6f
Merge pull request #405 from jfhs/jfhs/support-all-html-entities 
Improve HTML entities support
 2021-03-31 01:44:54 +02:00
Michael Christen
b2af745dd6
Merge pull request #404 from lnceballosz/master 
NGI0 - Updating licensing aspects according REUSE
 2021-03-30 23:48:21 +02:00
jfhs
10bddc2c2d Decode HTML entities in all property values by default 2021-03-30 22:24:55 +02:00
jfhs
2135d259e3 Replace hardcoded html/xml entities with a file, support decoding all defined HTML entities 2021-03-30 22:24:54 +02:00
Michael Peter Christen
8f876a8c72 added concurrency to enhance indexing speed during json surrogate import 2021-03-30 12:07:36 +02:00
Michael Peter Christen
f8cbaeef93 Merge branch 'master' of git@github.com:yacy/yacy_search_server.git 2021-03-29 18:46:53 +02:00
Michael Peter Christen
a857e3d3d5 fix for json importer 2021-03-29 18:46:42 +02:00
sgaebel
1546232c94 adds ranking for multi document queries only 2021-03-20 17:48:35 +01:00
sgaebel
93b353d22d does not boost or add fields for zero-row-queries (exists()) 2021-03-20 17:48:26 +01:00
sgaebel
f16cd154f7 removes unused imports and variables 2021-03-20 15:14:09 +01:00
sgaebel
c69c462a15 replaces a expensive getLoadTimeURL() by exists() 
refactors urlExists to getHarvestProcess as that is what it does
 2021-03-20 15:01:31 +01:00
sgaebel
a5488ac8f5 uses edismax queries on query counts > 1 only 2021-03-20 01:06:09 +01:00
sgaebel
26223dc25a replaces getLoadTime() by exists() with a simpler query 
since solr-8.8.1 getLoadTime() causes a high cpu usage
 2021-03-20 01:06:02 +01:00
sgaebel
8e4d014c06 removes useless SolrRequestInfo.clearRequestInfo(), avoids spamming the 
log
 2021-03-18 22:33:39 +01:00
Lina Ceballos
a96752f5ab adding SPDX license and copyright headers 2021-03-11 12:17:11 +01:00
Michael Peter Christen
e18d0ef544 trying to set a higher priority to the process that is involved in index 
export
 2021-03-09 00:04:05 +01:00
Michael Peter Christen
8b4394a6c5 fixes for solr 8.8.1 migration 
- replace new guava 30 with older 25 because that is the correct
dependency for solr 8.8.1. The newer one did actually not work!
- index will be crated in a DATA/INDEX/freeworld/SEGMENTS/solr_8_8_1
subfolder. The older solr_6_6 index is not touched but also not
migrated. The index starts with fresh (empty) content.
- Older indexes must be migrated by hand (export/import) so far until a
better solution is found.
- Large schema adoptions for lucene 8.8.1
 2021-03-08 13:39:27 +01:00
Michael Peter Christen
ed9789214e fixed seed initialization problem 2021-03-06 13:35:46 +01:00
Al Sutton
8ade8b8775 Remove forced clear to match new behaviour in 2da71c2a40 2021-03-04 16:37:56 +00:00
Al Sutton
09695fc6d3 Update exceptions to match updated API 2021-03-04 16:34:02 +00:00
Al Sutton
69014a701e Update API Usage 2021-03-04 16:14:56 +00:00
Michael Peter Christen
3da7628117 use environment variables to overwrite configuration variables 
you can i.e. do:
export YACY_PORT=8092 && ./startYACY.sh
Just append "YACY_" to uppercase version of environment variables and
replace all "." with "_".
 2021-02-09 20:26:49 +01:00
Michael Peter Christen
13a2e6dc6e Merge branch 'master' of https://github.com/yacy/yacy_search_server.git 2021-01-25 11:49:32 +01:00