Commit Graph

10328 Commits

Author SHA1 Message Date
reger
f09dbbef96 make SecurityHandler webappcontext ready 2014-01-10 12:36:42 +01:00
Michael Peter Christen
ca3411d805 added checkindex, solr index check 2014-01-10 12:27:49 +01:00
Michael Peter Christen
36594d0348 Merge branch 'master' of ssh://git@gitorious.org/yacy/rc1.git 2014-01-10 12:15:13 +01:00
reger
37f2a82a5d making root context (htroot) a WebAppContext
- this allows additional features, like servlet configuration via web.xml and many more things.
- currently the standard servlets are still configured in the code (so the supplied defaults/web.xml is not realy needed, yet),
  but could be expanded
- lookup for web.xml - 1. in /DATA/SETTINGS then in /defaults
2014-01-10 10:42:47 +01:00
reger
f6099b730d disabled unused fields in default Solr collection schema 2014-01-10 10:26:45 +01:00
reger
28eae57e8b spend CrawlQueues a fremem routine
- clears errorStack
- will not get hit often (but better little than nothing on low mem)
2014-01-10 10:24:33 +01:00
reger
b931bf6b48 fix use of url proxy access pattern
pattern of transparent was used.
2014-01-08 08:12:56 +01:00
reger
280c4a3ac1 exclude terms with " for didYouMean suggestion
causes Solr error (and wordindex likely finds suggestion)

org.apache.solr.core.SolrCore org.apache.solr.common.SolrException: org.apache.solr.search.SyntaxError: Cannot parse 'text_t:""d"': Lexical error at line 1, column 12.  Encountered: <EOF> after : ""
	at org.apache.solr.handler.component.QueryComponent.prepare(QueryComponent.java:171)
	at org.apache.solr.handler.component.SearchHandler.handleRequestBody(SearchHandler.java:187)
	at org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:135)
	at net.yacy.cora.federate.solr.connector.EmbeddedSolrConnector.query(EmbeddedSolrConnector.java:179)
	at net.yacy.cora.federate.solr.connector.EmbeddedSolrConnector$DocListSearcher.<init>(EmbeddedSolrConnector.java:345)
	at net.yacy.cora.federate.solr.connector.EmbeddedSolrConnector.getCountByQuery(EmbeddedSolrConnector.java:364)
	at net.yacy.cora.federate.solr.connector.MirrorSolrConnector.getCountByQuery(MirrorSolrConnector.java:326)
	at net.yacy.cora.federate.solr.connector.ConcurrentUpdateSolrConnector.getCountByQuery(ConcurrentUpdateSolrConnector.java:440)
	at net.yacy.search.index.Segment.getWordCountGuess(Segment.java:464)
	at net.yacy.data.DidYouMean.getSuggestions(DidYouMean.java:181)
	at suggest.respond(suggest.java:73)
2014-01-08 04:46:21 +01:00
reger
fbc1071f6d Merge origin/master 2014-01-07 22:48:45 +01:00
reger
7b800a0c8e fix: NPE on shutdown via script 2014-01-07 22:44:24 +01:00
Michael Peter Christen
bfaf42b0b8 added a script which can check the solr index for inconsistencies while
the peer is down. This shall be used in emergency cases where a check or
fix for a broken solr index is needed.
2014-01-07 21:58:55 +01:00
Michael Peter Christen
ce4d42d77c Merge branch 'master' of ssh://git@gitorious.org/yacy/rc1.git 2014-01-07 21:52:38 +01:00
Michael Peter Christen
644573cfc4 using the adminAccountUserName from yacy.conf within apicall.sh 2014-01-07 21:52:19 +01:00
reger
6932aa4d7a use configured admin-username for api calls
- the admin user name can be configured, in apiExec calls the default "admin" username is used. 

TODO: the bin/apicall.sh script should likely take that into account.
2014-01-07 21:26:50 +01:00
reger
c656e67c97 fix: display proper error msg on admin user change 2014-01-07 20:34:37 +01:00
orbiter
2ead4e44d9 introduced a new storage path ARCHIVE inside of DATA which will be used
as path for solr index dumps (instead of the SEGMENTS path). This will
make a maintenance of index backups easier. It will also provide a tool
to migrate from an freeworld index to a webportal index.
2014-01-07 17:53:49 +01:00
sixcooler
add0e42804 fix double-escaped urls from proxy-usage 2014-01-07 01:04:33 +01:00
sixcooler
865ce6f974 check blacklist proxyClient config 2014-01-07 01:01:55 +01:00
sixcooler
345f9aba27 make use of our DNS-cache again - this realy speeds up the lookup 2014-01-07 00:18:01 +01:00
reger
e6d284fe1e better solution for prev. commit with MultiMapSolrParams.getFieldInt not returning default parameter 2014-01-06 18:19:54 +01:00
reger
0bc2fc14ab improve NPE chance on missing parameters
java.lang.NullPointerException
	at net.yacy.http.servlets.SolrServlet.service(SolrServlet.java:145)
	at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:684)
	at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:501)
2014-01-06 17:52:21 +01:00
reger
f06cef5d5b reimplement proxy access by configured whitlist pattern
was currently limited to own ip.
2014-01-06 15:00:14 +01:00
reger
05d6cc6ea3 setting of IPv4Stack moved earlier
it seems even better to call system.setproperty before isrunning check
(if nothing helps we have to set it in startup script)
2014-01-06 11:28:05 +01:00
reger
1b6d173b14 update to Jetty 8.1.14 2014-01-06 08:48:43 +01:00
reger
41dc0f82c1 open service manager upon install failure in installYaCyWindowsService.bat
likely service is already installed
Service Manager allows then to modify settings
2014-01-06 07:22:23 +01:00
reger
30d925a96e reimplemented server access restriction
via Jetty IPAccessHandler to allow only configured IP's to access.
Handler is only loaded if a restriction is configured.

Since IPAcessHandler (Jetty 8) does not support IPv6 system property java.net.preferIPv4Stack=true
Testing showed system.setProperty seems to be sensitive to point of calling (earliest possible time seems to be best = early in yacy.main).
Moved the "isrunning..." just open browser check also to the new routine to preread the yacy.config only once.
2014-01-06 07:00:16 +01:00
orbiter
3cb6c7861f fixed shutdown authenticaton problem 2014-01-06 01:48:54 +01:00
Michael Peter Christen
ed06b5b94b set a realm message to log-in input window which explains that a
password for the account 'admin' can be (re-)set with the script
bin/passwd.sh
2014-01-05 17:43:34 +01:00
Michael Peter Christen
7005ecdabd cleanup 2014-01-05 15:06:40 +01:00
Michael Peter Christen
2939b47986 removed non-working realm setting in http client (auth for localhost was
added in previous commit)
2014-01-05 15:04:18 +01:00
orbiter
9d52b337f3 added http authentification to YaCy http client for all localhost
acesses to enable self-steering of the peer using the API table. This is
necessary in case that an password for the administration pages is set.
2014-01-05 14:46:11 +01:00
Michael Peter Christen
c951945666 modified log-in detail to enable admin-login from localhost with stored
hash even if localhost access is disabled. This is urgently needed for
the apicall.sh script since that is used for high-availability set-up
(checkalive and indexdump for index mirroring)
2014-01-05 11:50:23 +01:00
Michael Peter Christen
9bd71fdbb4 made the access tracker class static because it shall be used by the
jetty auth module
2014-01-05 05:04:28 +01:00
Michael Peter Christen
1c56befb93 fixed mess with test on localhost (which means local hosts for some
cases)
2014-01-05 04:55:30 +01:00
Michael Peter Christen
7d6fc79eb8 refactoring (usage of constant names for attributes of authentication
check)
2014-01-05 04:23:44 +01:00
Michael Peter Christen
b9d36e45e0 removed the &amp explicit encoding of ampersand character since this is
double-translated within the template replacement process.
2014-01-05 03:40:10 +01:00
reger
e2ccb6ce9d modified DefaultServlet parameter on invoke templates
call response with post=0 (if post empty) simulating previous behavior.

(template servlets typically test for post==null,
found one more Crawler.p.java were empty post caused problem,
= defaults not correctly set)
2014-01-04 20:49:26 +01:00
reger
cabe0943cd fix opensearch resultcount in yacysearch.rss
see merge request https://gitorious.org/yacy/rc1/merge_requests/24
use result count in searchtrailer.xml which is on p2p search more accurate (timing)
2014-01-04 17:14:10 +01:00
reger
eaf596a257 adding proxy status to (private) status box
(show also transparent and url proxy status)

show search result via url proxy only if status=on
2014-01-04 16:10:54 +01:00
reger
4c38bceafc handle http connect for proxy
refactor header cleanup (reuse existing code)
2014-01-04 13:09:34 +01:00
reger
cfabe8f67a harmonize access restriction for urlproxy servlet
with proxy handler, what is currently
- use switched on in config
- access from a local IP / hostname

fix shutdown exception for crashprotection handler on interrupted connections.
2014-01-03 12:28:40 +01:00
reger
e3d8459906 extend ssl enabled msg on status page
- post the portnr
2014-01-03 02:56:09 +01:00
reger
e6b9643fd6 extended request for local peer check to by hostname resolved ip
the current islocal() check did not detect a domain.com address as request for the local peer.
2014-01-03 01:13:56 +01:00
reger
c797f108a1 add error response on deniedl proxy access
send http 403 response
2014-01-02 09:11:08 +01:00
reger
0583f44306 reimplement proxy access log (to Jetty ProxyHandler)
- using existing HTTPDProxyHandler logger
- allow local loopback ip to access proxy
2014-01-02 03:37:33 +01:00
reger
8cbc1c970a Security Hot-Fix: for transparent proxy. 2014-01-01 20:48:35 +01:00
reger
58ecf5e4dd add to blacklist button in CrawlResults
http://bugs.yacy.net/view.php?id=220
introduced Blacklist.add with sourcefile only parameter
2014-01-01 11:01:22 +01:00
reger
17b454f957 fix external link (open in new tab) 2014-01-01 10:33:20 +01:00
reger
e9081c0f17 moved startup execAPIActions call after Jetty startup
execAPIActions require http to be up. The 10s sleep was sufficient to allow Jetty to start, 
but it's more robust to place the call after http is assigned to switchboard/serverSwitch.
2014-01-01 10:28:49 +01:00
reger
19c1a7a5ca change SolrServlet from Filter to Servlet
(as no multicore required)
this allows to simplify context/servlet initialization in Jetty init.
2014-01-01 10:20:32 +01:00