Marown/yacy_search_server
1
0
Fork 0
mirror of https://github.com/yacy/yacy_search_server.git synced 2024-09-21 00:00:13 +02:00
Code Issues Actions Packages Projects Releases Wiki Activity
10,572 Commits 1 Branch 20 Tags 836 MiB
f8e998d660
Commit Graph

2537 Commits

Author SHA1 Message Date
orbiter
7d24bcb98d added flag to require that all web pages, even such without a "_p" 
extension require authorization. (default off)
 2014-03-20 19:09:47 +01:00
Michael Peter Christen
7a6658abec removed synchronization in embedded solr connection (that was probably 
a mistake?)
 2014-03-19 16:21:03 +01:00
Michael Peter Christen
a7d4379ef9 fixed shutdown of solr cores in case that more than one local core is to 
be closed (this happens if webgraph is enabled and the index is dumped
using /IndexControlURLs_p.html
 2014-03-19 12:23:40 +01:00
Michael Peter Christen
453bfd0f17 removed unused variables and warnings 2014-03-19 09:29:01 +01:00
Michael Peter Christen
05655d98df Merge branch 'master' of ssh://git@gitorious.org/yacy/rc1.git 2014-03-17 11:57:01 +01:00
reger
9f02d2c47b fix: remove link to triplestore in Vocabulary_p (triplestore does not longer exist) 
- should be investigated in more detail to look for additional implications
Remove "yacyaction" from proxyservlet as it was only needed for removed interaction routines.
 2014-03-16 22:11:19 +01:00
reger
81a846ec33 fix: set YaCy CONNECTION_PROP_HOST Header in ProxyServlet to host incl. port 2014-03-16 20:51:32 +01:00
reger
251be9ecfa remove unused ProxySettings ref. from loader 
clean unused whois test code
 2014-03-16 05:19:01 +01:00
reger
82dc815af9 cleanup: remove unrelated and unused code 2014-03-16 00:15:12 +01:00
Michael Peter Christen
85a427ec54 support for multiple sitemaps in robots.txt 2014-03-14 13:33:23 +01:00
reger
a373fb717d remove more unused from legacy server.http 
- triggerOnlineAction not used
- useTemplateCache not used
 2014-03-14 03:12:04 +01:00
reger
749d020aeb remove redundant url string manipulation in HTTPDProxyHandler 
(still used by ProxyServlet)
 2014-03-14 02:24:12 +01:00
reger
612294cf84 use servletPath in ProxyServlet instead of fixed name 
to allow servlet-mapping via web.xml
 2014-03-13 02:46:05 +01:00
reger
1d01672bd3 fix DCEntry.getIdentifier 
on successful url parameter
 2014-03-12 23:35:57 +01:00
Michael Peter Christen
b08375da33 fix for bad/missing values of size_i 2014-03-11 09:51:04 +01:00
reger
6306d28a6a OAI import get multivalued keywords (dc:subject) 2014-03-09 03:15:35 +01:00
reger
0a8c8102de allow YaCy to start w/o ssl if JKS init fails 2014-03-07 20:06:14 +01:00
sixcooler
0b2101c59c Speed up the ProxyHandler: 
simplified cache-storing and make it concurrent in order to free the
clientconnection asap
let other prozesses wait on proxy-access like it was bevore
 2014-03-07 17:47:09 +01:00
reger
516f8c2489 fix: to allow unix scripts (bin/*.sh) to allways submit http admin apicalls 
using auth via config hash (legacy requirement)
 2014-03-07 00:16:57 +01:00
Michael Peter Christen
ea3aa30593 Merge branch 'master' of ssh://git@gitorious.org/yacy/rc1.git 2014-03-06 03:33:33 +01:00
reger
dd5bf0b71b cleanup old reference to HTTPDemon.setAlternativeResolver 
optimize .yacyh check in AbstractRemoteHandler
 2014-03-06 03:08:04 +01:00
Michael Peter Christen
51800007c4 - added concurrency to postprocessing of webgraph document 
- bundeled separate webgraph postprocesing steps into one
 2014-03-06 01:43:48 +01:00
Michael Peter Christen
5f4a6892c1 enhanced RowSet re-sort limit for small sets 2014-03-05 23:28:19 +01:00
reger
351c2be68d fix: make sure adminAccount changes made via ConfigAccounts_p are effective immediately 
force to remove current credentials from knownuser cache
 2014-03-05 02:59:27 +01:00
reger
5c9dcc269d improve OAI-PMH import identifier recognition 
- find best fittng identifier (url) by checking all given dc:identifier in record (many entries proviede several identifiers)
  as identifier is currently a multivalued field use "getParams" in preference of splitting the 1st string by ";" 
- add resolve DOI:... identifier via http://dx.doi.org/
 2014-03-04 03:08:37 +01:00
Michael Peter Christen
0e7d249a69 fixed another shutdown problem (only occurs if webgraph core is enabled) 2014-03-04 01:36:38 +01:00
Michael Peter Christen
e485fbd0ce - let crawl loader jobs die after 10 seconds without new jobs 
- corrected shutdown order t prevent a deadlock during shutdown
 2014-03-04 00:33:13 +01:00
Michael Peter Christen
bcd9dd9e1d enhanced concurrent loading by using a fixed set of concurrent loader 
processes in favor of throwaway-processes. The control mechanism does
less often report a 'queue full' message to the busy loop which then
does not perform a long busy waiting; instead all requests are queued
and new loader processes are started if necessary up to a given limit
(as set before)
 2014-03-03 22:13:40 +01:00
orbiter
051328271c bugfix-bugfix 2014-03-02 21:13:38 +01:00
orbiter
eedcbcd906 bugfix to proxy handler: recognize the own yacyh-host 2014-03-02 12:10:19 +01:00
orbiter
d68e5ad0c4 NPE fix for Thread name (just commited yesterday, sorry) 2014-03-02 11:20:48 +01:00
reger
6878c90f99 fix: IPv6 INTRANET_PATTERNS for local ip (see http://bugs.yacy.net/view.php?id=378) 
requiring following ":" for fc and fd prefix and made pattern match case insesitive
- add some more ipv6 test cases to MultiProtocolURLTest.java
 2014-03-02 06:13:21 +01:00
reger
a2e5ea2026 status panel link to set max mem 
+url proxy same error text as in transparent
 2014-03-01 00:56:45 +01:00
Michael Peter Christen
6ed9c0164e attaching names to all Threads to get a better view in profiling tools 
like VisualVM
 2014-02-28 15:02:01 +01:00
Michael Peter Christen
fdaeac374a - enhanced postprocessing speed and memory footprint (by using HashMaps 
instead of TreeMaps)
- enhanced memory footprint of database indexes (by introduction of
optimize calls)
- optimize calls shrink the amount of used memory for index sets if they
are not changed afterwards any more
 2014-02-28 14:01:09 +01:00
reger
ba49ff81ed little more verbose proxy 403 error message 2014-02-28 03:14:07 +01:00
Michael Peter Christen
d325cb8912 fixes and enhancements for postprocessing 2014-02-28 02:51:14 +01:00
Michael Peter Christen
7c1b968378 another fix for the shutdown exceptions 2014-02-28 01:53:32 +01:00
orbiter
133d41386c (again) full redesign of ConcurrentUpdateSolrConnector to remove 
out-of-order transactions regarding add and delete operations. Now all
operations (add and delete) are executed concurrently in-order.
 2014-02-28 00:19:30 +01:00
Michael Peter Christen
a632b0d2a4 added a forced commit to index deletion to enable synchronized index 
updates
 2014-02-27 12:50:40 +01:00
Michael Peter Christen
1d069c5861 make sure that postprocessed documents are overwritten 2014-02-27 12:27:15 +01:00
Michael Peter Christen
0d2342575e Merge branch 'master' of ssh://gitorious.org/yacy/rc1 2014-02-27 01:29:52 +01:00
Michael Peter Christen
3cc5c0ffdd a concurrency enhancement which was not used because tests showed worse 
indexing speed. I leave the code there since it may be useful in
SolrCloud environments.
 2014-02-27 01:27:06 +01:00
Michael Peter Christen
e644981697 added one more postprocessing low memory check 2014-02-27 00:34:13 +01:00
reger
5e645f4449 Merge origin/master 2014-02-27 00:24:30 +01:00
reger
3b89176b9f use config value htroot in Jetty init (was hardcoded) 
- move htroot exist check from old httpdfilehandler to startup, remove from filehandler and legacy proxyhandler
- use SwitchboardConstant.htroot where appropriate
 2014-02-27 00:23:34 +01:00
Michael Peter Christen
e1bf65c892 added short memory protection during postprocessing 2014-02-26 23:02:56 +01:00
Michael Peter Christen
90b47e83e6 fixed shutdown error when closing solr connectors 2014-02-26 22:47:16 +01:00
Michael Peter Christen
7640834b37 removed double concurrency to put Solr documents into the index. The 
writings to the solr index are also buffered in
ConcurrentUpdateSolrConnector
 2014-02-26 22:21:00 +01:00
Michael Peter Christen
0f6b72f24b do not use luke requests for remote solr servers if the result is 
different from normal requests. This happens if the remote solr is
actually a solrCloud; in such cases the luke request returns only the
result of the single solr peer, not the whole cloud.
also done: some refactoring.
 2014-02-26 14:30:48 +01:00