Commit Graph

10444 Commits

Author SHA1 Message Date
reger
9b24dae2b7 add language navigation filter clause to rwi results 2014-01-25 22:59:23 +01:00
reger
f307d65dcf prepare for a language navigator
works fine to restrict language for local solrSearches.
More work needs to be done to make rwi/remote searches respect the modifier.language restriction.
2014-01-24 03:11:25 +01:00
reger
cf553e5045 added hint to web.xml and for completeness the full set of hardcoded mappings 2014-01-23 23:56:45 +01:00
orbiter
768b1306b8 Added a write-enabled checkbox for remote solr servers.
It is now possible to assign every peer other YaCy peers as remote solr
server which are only used for read operations during search. This also
affects crawling: it will exclude urls from crawls which exist on remote
solr/remote YaCy peers.
2014-01-23 22:48:31 +01:00
orbiter
f7d6dd136f changed solr paths according to new default paths 2014-01-23 19:21:07 +01:00
Michael Peter Christen
c84bcc878a first try to add a generic solr servlet as luke request servlet 2014-01-23 19:01:31 +01:00
Michael Peter Christen
a8fdaace31 changed the web.xml as well to migrate the solr servlet 2014-01-23 18:41:45 +01:00
Michael Peter Christen
4cb7e2a2ca refactoring: renamed the SolrServlet to SolrSelectServlet for better
naming of more Solr Servlets
2014-01-23 17:20:49 +01:00
Michael Peter Christen
dc06e407ce added two virtual instances of solr for the both cores: collection1 and
webgraph. These cores are now accessible at
/solr/collection1/select instead /solr/select?core=collection1
and
/solr/webgraph/select instead /solr/select?core=webgraph
in addition to the old behavior to support compatibility to the old
peers. These new paths are fully solr standard-conform and will allow
the cross-linking between YaCy peers using their public solr API.
2014-01-23 17:14:13 +01:00
Michael Peter Christen
8b14e92ba4 added button in host browser to re-load 404/failed documents 2014-01-23 15:56:36 +01:00
reger
f47067b0ce fix search navigator not showing activated nav
introduced with 97e84439fb
2014-01-23 01:52:51 +01:00
orbiter
771d8261c1 Merge branch 'master' of ssh://git@gitorious.org/yacy/rc1.git 2014-01-22 21:53:27 +01:00
orbiter
c351e47a84 fix for bad-formatted lonlat 2014-01-22 21:33:11 +01:00
reger
4c603b216e optimize parse ServerSideInclude 2014-01-22 21:23:32 +01:00
orbiter
5ec0c969c9 fix for http://bugs.yacy.net/view.php?id=354 2014-01-22 20:59:53 +01:00
orbiter
0002abd583 fix for OOM during remote search and too high load protection 2014-01-22 20:54:03 +01:00
sixcooler
5a917e13c6 use less ram on dht-URL transfer by not using a URIMetadataNode[] 2014-01-22 17:52:07 +01:00
Michael Peter Christen
c87cdfca2e do not set a load prerequisite that prevents the start of one-time-jobs 2014-01-22 17:18:53 +01:00
sixcooler
0512e46c6a bump to httpclient-4.3.2 2014-01-22 01:31:22 +01:00
sixcooler
4d77ca52c9 workaround to let dht-out run on smal Systems like a Pi 2014-01-22 01:26:44 +01:00
reger
9a96a7d73f put list quick navigator buttons belowon BlackList_p editor
replacing the dropdown -> go navigation
2014-01-21 21:35:48 +01:00
Michael Peter Christen
6ada0daae9 making latency_factor and maximum number of same hosts in loader queue
settings available in Crawler_p.html servlet for steering.
2014-01-21 19:28:00 +01:00
Michael Peter Christen
489c3fbc90 code simplifications / removed warnings 2014-01-21 17:53:39 +01:00
Michael Peter Christen
0168f80c28 new crawling factors can now be changed during runtime 2014-01-21 17:52:16 +01:00
Michael Peter Christen
be5e808236 - removed hardcoded load-test which is now handled in BusyQueues
steering, see /PerformanceQueues_p.html
- changed default values for crawler queue load limit (high, because
these jobs are started upon user request)
2014-01-21 17:48:45 +01:00
sixcooler
40a4030b55 configurable max-load values for YaCy-Threads:
try lower values on smal systems like a Pi
2014-01-21 17:04:22 +01:00
sixcooler
6d8c023a5e lower client-connection for single-cpu-systems 2014-01-21 16:56:44 +01:00
Michael Peter Christen
77531850b5 reverted crawling strategy from latest commit. 2014-01-21 16:05:55 +01:00
Michael Peter Christen
c0da966dfa enhanced crawler speed 2014-01-20 21:46:40 +01:00
Michael Peter Christen
79809342fa added synchronization to exists() call bacause the concurrent call to
that method showed in thread dump close to deadlock situations. Its also
better to synchronize IO operations because they become faster then.
2014-01-20 21:09:03 +01:00
Michael Peter Christen
9a6912f2e6 if a http client thread is still running but we do not wait for it any
more, call an interrupt
2014-01-20 18:39:36 +01:00
Michael Peter Christen
0d235a565b cleanup crawl loader jobs 2014-01-20 18:36:00 +01:00
Michael Peter Christen
1ea17bd9f3 - removed old metadata database and all migration code
- refactored all code which uses URIMetadataRow as standard for word
hash length and word hash ordering and moved that to the class 'Word',
becuase the class URIMetadataRow defined the old metadata data structure
and should be superfluous in the future
- removed unused methods from URIMetadataRow as preparation for further
removal of that class
2014-01-20 18:31:46 +01:00
reger
d3de309953 fix IOexception logging issue in DefaultServlet
reason not sure but .logException triggers another exception
2014-01-20 08:12:35 +01:00
reger
97e84439fb adjusted ConfigHeuristic and changed QueryGoal.getOriginalQueryString to .getQueryString
- since specific heuristic Twitter & Blekko is not longer available or redundant with OpenSearchHeuristic,
adjusted ConfigHeuristic to use OpensearchHeuristic settings only.
For this the default OSD search target list is made available (copied) by default and the other configs are removed.

- the return of QueryGoal.getOriginalQueryString includes the queryModifier, which are held separately in a modifier object,
but in most (all) cases just the query term is expected, clarified and renamed it to QueryGoal.getQueryString which returns
just the search term (if needed a .getOrigianlQueryString could be implemented in Queryparameters, adding the modifiers)

- started to adjust internal html href references from absolute to relative (currently it is mixed).
For future development we should prefer relative href targets (less trouble with context aware  servlets)
2014-01-20 00:58:17 +01:00
reger
d24a0ec32c upd heuristic default list (heuristicopensearch.conf)
- Faroo Web taken out (requires api key) http://www.faroo.com/hp/api/api.html#description
- update Faroo News to new url
- Twitter taken out (change to Api 1.1 not supporting rss) https://dev.twitter.com/discussions/24239
2014-01-20 00:03:55 +01:00
Michael Peter Christen
022c6d3ce1 do YaCy p2p connections using a timeout-request which covers the http
request into a separate thread and ignores the furthure result of a
request if that does not answer within the requested time-out. This is a
try to solve a problem with the peer-ping, which hangs whenever a peer
appears to be dead or blocked.
2014-01-19 15:21:23 +01:00
Michael Peter Christen
42f3733a05 Merge branch 'master' of ssh://git@gitorious.org/yacy/rc1.git 2014-01-19 14:47:24 +01:00
Michael Peter Christen
25a6c05008 experimental removal of synchronization. This should work for all cases
where the size() and isEmpty() method is used only for statistics, which
happens at many locations in YaCy. If these methods are used for
structual reasons (like accessing the last element in an array) then it
may fail or cause other problems. As far as visible, this is not the
case.
2014-01-19 14:47:11 +01:00
Michael Peter Christen
5695280edd removed superfluous synchronization 2014-01-19 14:44:58 +01:00
Michael Peter Christen
a1977b7a75 removed debug code 2014-01-19 14:42:26 +01:00
orbiter
fd4abc0565 Merge branch 'master' of ssh://git@gitorious.org/yacy/rc1.git 2014-01-19 01:50:55 +01:00
orbiter
d5b8e473c8 added load limit for DHT transfer: RWI acceptance only if local load is
not too high
2014-01-19 01:50:42 +01:00
reger
41c126978b fix bug: Crawl Start (Expert) crawls "?-URLs" even if told not to do so
http://bugs.yacy.net/view.php?id=329
2014-01-18 23:27:16 +01:00
reger
2614fa7aeb Skip remote Solr search if last try showed error
As the solr servlet may not be available (e.g. no public search page, old version, individual access setting) a /solr/select error is 
remembered in the seed.dna of the remote peer.
This is not permanent, as flag is not stored and the seed is reloaded on several occasions, it is just a memory of the recent past status.
Might also be set to "not available" on time-out of last try.
2014-01-18 18:48:52 +01:00
orbiter
a07e9b3582 concurrency-solid version of transmission limitation 2014-01-18 12:55:05 +01:00
orbiter
ec21f0494e removed -d64 jvm option because that causes problems on non-64 bit
linux, see http://bugs.yacy.net/view.php?id=349 and
http://bugs.yacy.net/view.php?id=339
2014-01-18 12:54:14 +01:00
orbiter
60ead31273 Merge branch 'master' of ssh://git@gitorious.org/yacy/rc1.git 2014-01-18 10:50:36 +01:00
orbiter
52bf7d1ac8 reduce load during dht transfer 2014-01-18 10:50:24 +01:00
sixcooler
f0587d4af5 NP-fix, which was found on a Pi under 'havy' load 2014-01-18 00:03:44 +01:00