Michael Peter Christen
e485fbd0ce
- let crawl loader jobs die after 10 seconds without new jobs
...
- corrected shutdown order t prevent a deadlock during shutdown
2014-03-04 00:33:13 +01:00
Michael Peter Christen
bcd9dd9e1d
enhanced concurrent loading by using a fixed set of concurrent loader
...
processes in favor of throwaway-processes. The control mechanism does
less often report a 'queue full' message to the busy loop which then
does not perform a long busy waiting; instead all requests are queued
and new loader processes are started if necessary up to a given limit
(as set before)
2014-03-03 22:13:40 +01:00
orbiter
051328271c
bugfix-bugfix
2014-03-02 21:13:38 +01:00
orbiter
eedcbcd906
bugfix to proxy handler: recognize the own yacyh-host
2014-03-02 12:10:19 +01:00
orbiter
d68e5ad0c4
NPE fix for Thread name (just commited yesterday, sorry)
2014-03-02 11:20:48 +01:00
reger
6878c90f99
fix: IPv6 INTRANET_PATTERNS for local ip (see http://bugs.yacy.net/view.php?id=378 )
...
requiring following ":" for fc and fd prefix and made pattern match case insesitive
- add some more ipv6 test cases to MultiProtocolURLTest.java
2014-03-02 06:13:21 +01:00
reger
a2e5ea2026
status panel link to set max mem
...
+url proxy same error text as in transparent
2014-03-01 00:56:45 +01:00
Michael Peter Christen
6ed9c0164e
attaching names to all Threads to get a better view in profiling tools
...
like VisualVM
2014-02-28 15:02:01 +01:00
Michael Peter Christen
fdaeac374a
- enhanced postprocessing speed and memory footprint (by using HashMaps
...
instead of TreeMaps)
- enhanced memory footprint of database indexes (by introduction of
optimize calls)
- optimize calls shrink the amount of used memory for index sets if they
are not changed afterwards any more
2014-02-28 14:01:09 +01:00
reger
ba49ff81ed
little more verbose proxy 403 error message
2014-02-28 03:14:07 +01:00
Michael Peter Christen
d325cb8912
fixes and enhancements for postprocessing
2014-02-28 02:51:14 +01:00
Michael Peter Christen
7c1b968378
another fix for the shutdown exceptions
2014-02-28 01:53:32 +01:00
orbiter
133d41386c
(again) full redesign of ConcurrentUpdateSolrConnector to remove
...
out-of-order transactions regarding add and delete operations. Now all
operations (add and delete) are executed concurrently in-order.
2014-02-28 00:19:30 +01:00
Michael Peter Christen
a632b0d2a4
added a forced commit to index deletion to enable synchronized index
...
updates
2014-02-27 12:50:40 +01:00
Michael Peter Christen
1d069c5861
make sure that postprocessed documents are overwritten
2014-02-27 12:27:15 +01:00
Michael Peter Christen
0d2342575e
Merge branch 'master' of ssh://gitorious.org/yacy/rc1
2014-02-27 01:29:52 +01:00
Michael Peter Christen
3cc5c0ffdd
a concurrency enhancement which was not used because tests showed worse
...
indexing speed. I leave the code there since it may be useful in
SolrCloud environments.
2014-02-27 01:27:06 +01:00
Michael Peter Christen
e644981697
added one more postprocessing low memory check
2014-02-27 00:34:13 +01:00
reger
5e645f4449
Merge origin/master
2014-02-27 00:24:30 +01:00
reger
3b89176b9f
use config value htroot in Jetty init (was hardcoded)
...
- move htroot exist check from old httpdfilehandler to startup, remove from filehandler and legacy proxyhandler
- use SwitchboardConstant.htroot where appropriate
2014-02-27 00:23:34 +01:00
Michael Peter Christen
e1bf65c892
added short memory protection during postprocessing
2014-02-26 23:02:56 +01:00
Michael Peter Christen
90b47e83e6
fixed shutdown error when closing solr connectors
2014-02-26 22:47:16 +01:00
Michael Peter Christen
7640834b37
removed double concurrency to put Solr documents into the index. The
...
writings to the solr index are also buffered in
ConcurrentUpdateSolrConnector
2014-02-26 22:21:00 +01:00
Michael Peter Christen
0f6b72f24b
do not use luke requests for remote solr servers if the result is
...
different from normal requests. This happens if the remote solr is
actually a solrCloud; in such cases the luke request returns only the
result of the single solr peer, not the whole cloud.
also done: some refactoring.
2014-02-26 14:30:48 +01:00
Michael Peter Christen
c57026e242
recover from OOM
2014-02-25 15:23:45 +01:00
Michael Peter Christen
907db8b7a6
fix for bad query shortcut hack
2014-02-25 15:19:04 +01:00
Michael Peter Christen
a2b66fe2eb
Merge branch 'master' of ssh://git@gitorious.org/yacy/rc1.git
2014-02-25 14:37:39 +01:00
Michael Peter Christen
9f6be762a6
- better logging for postprocessing
...
- fixed collection bug in postprocessing
2014-02-25 14:37:30 +01:00
orbiter
da5d4128bf
prevent npe
2014-02-25 03:26:20 +01:00
orbiter
a878c7982c
prevent npe
2014-02-25 03:19:41 +01:00
orbiter
e4eb87d924
Merge branch 'master' of ssh://git@gitorious.org/yacy/rc1.git
2014-02-25 02:16:37 +01:00
orbiter
ced1a96f9c
fixed error cache
2014-02-25 02:16:22 +01:00
reger
3ba81bd08a
Merge origin/master
2014-02-25 00:24:10 +01:00
reger
4d896383db
fix: use timeout = proxy.ClientTimeout in ProxyHandler
...
(was 10sec fix) see http://bugs.yacy.net/view.php?id=236
2014-02-25 00:23:06 +01:00
orbiter
cfb647db6e
- introduced a miss cache in ConcurrentUpdateSolrConnector
...
- better usage of cache
- bugfix for postprocessing
2014-02-24 23:42:50 +01:00
orbiter
a87d8e4a8e
changed caching of ConcurrentUpdateSolrConnector: it caches now also the
...
url along with the load date. While this takes much more memory, it
eliminates database lookups for getURL() requests, which happen equally
often. This speeds up remote solr configurations.
2014-02-24 22:59:58 +01:00
orbiter
f6e441dd77
refactoring
2014-02-24 21:01:56 +01:00
orbiter
76c53faeb2
removed unused code (HostStat)
2014-02-24 20:51:43 +01:00
orbiter
d3a88eaecb
introducing ConcurrentUpdateSolrServer for remote solr servers.
...
Scaling of write buffers and update queue size is made according to
assigned memory.
2014-02-24 20:26:02 +01:00
reger
809e976578
remove unused java imports form yacy.java
2014-02-24 05:19:40 +01:00
reger
a9b06f8719
add a -config command line parameter e.g. -config "port=9090" "port.ssl=8043"
...
- useful for remote installation to set any config file property
- multipe parameter can be set at once, on Windows enclose parameter in doublequotes
- special handling "adminAccount=adminuser:adminpwd" sets adminusername and md5 encoded admin-pwd
- adjusted windows startbatch to allow command line parameter handling
- remove not needed classpath calculation from startYACY_debug.bat
2014-02-24 05:16:31 +01:00
reger
0923b09216
fix: allow 4 character admin user name
...
(was min 5 char)
2014-02-24 00:01:11 +01:00
Michael Peter Christen
254a7ac66c
fixed cleaning of index
2014-02-22 01:35:01 +01:00
Michael Peter Christen
28a7b42e6b
removed warning "sun.misc.BASE64Encoder is internal proprietary API and
...
may be removed in a future release"
2014-02-22 00:52:49 +01:00
Michael Peter Christen
046f5a03cb
one more SolrIndexSearcher bugfix
2014-02-21 23:48:56 +01:00
sixcooler
78c01b3eff
fix for 'AlreadyClosedException: this IndexReader is closed'
2014-02-21 17:28:32 +01:00
Michael Peter Christen
1b5e3d523a
better control over close-state of remote solr connections
2014-02-20 00:39:19 +01:00
Michael Peter Christen
1a364572a5
fix for
...
"org.apache.solr.core.SolrCore Too many close [count:-1] on
org.apache.solr.core.SolrCore@51af7c57"
-error
2014-02-20 00:03:35 +01:00
Michael Peter Christen
69391e5d9e
changed strategy to test existence of documents in Solr: using the
...
update time. The reason for that is a better caching for the crawler
double-check, which needs the update time for crawler steering.
2014-02-19 04:03:45 +01:00
Michael Peter Christen
790f103f32
delete fail-docs during postprocessing to prevent that they will appear
...
again and stay in postprocessing forever.
2014-02-18 01:38:56 +01:00
