Commit Graph

10281 Commits

Author SHA1 Message Date
reger
ea7cef5d05 fix NPE in TemplateEngine
StackTrace For input string: ""
java.lang.NumberFormatException: For input string: ""
	at java.lang.NumberFormatException.forInputString(NumberFormatException.java:65)
	at java.lang.Integer.parseInt(Integer.java:504)
	at java.lang.Integer.parseInt(Integer.java:527)
	at net.yacy.server.http.TemplateEngine.writeTemplate(TemplateEngine.java:241)
	at net.yacy.server.http.TemplateEngine.writeTemplate(TemplateEngine.java:199)
	at net.yacy.http.servlets.YaCyDefaultServlet.handleTemplate(YaCyDefaultServlet.java:896)
2014-01-10 18:11:32 +01:00
reger
cb6d0c2113 implementing YaCy legacy role names
- taking out customized SecurityHandler code as the original/default seems to just work fine
- with this individual sec. constraints can be applied via web.xml (using legacy role names)
2014-01-10 14:07:49 +01:00
reger
530b9f6de8 Merge origin/master 2014-01-10 12:38:00 +01:00
reger
f09dbbef96 make SecurityHandler webappcontext ready 2014-01-10 12:36:42 +01:00
Michael Peter Christen
ca3411d805 added checkindex, solr index check 2014-01-10 12:27:49 +01:00
Michael Peter Christen
36594d0348 Merge branch 'master' of ssh://git@gitorious.org/yacy/rc1.git 2014-01-10 12:15:13 +01:00
reger
37f2a82a5d making root context (htroot) a WebAppContext
- this allows additional features, like servlet configuration via web.xml and many more things.
- currently the standard servlets are still configured in the code (so the supplied defaults/web.xml is not realy needed, yet),
  but could be expanded
- lookup for web.xml - 1. in /DATA/SETTINGS then in /defaults
2014-01-10 10:42:47 +01:00
reger
f6099b730d disabled unused fields in default Solr collection schema 2014-01-10 10:26:45 +01:00
reger
28eae57e8b spend CrawlQueues a fremem routine
- clears errorStack
- will not get hit often (but better little than nothing on low mem)
2014-01-10 10:24:33 +01:00
reger
b931bf6b48 fix use of url proxy access pattern
pattern of transparent was used.
2014-01-08 08:12:56 +01:00
reger
280c4a3ac1 exclude terms with " for didYouMean suggestion
causes Solr error (and wordindex likely finds suggestion)

org.apache.solr.core.SolrCore org.apache.solr.common.SolrException: org.apache.solr.search.SyntaxError: Cannot parse 'text_t:""d"': Lexical error at line 1, column 12.  Encountered: <EOF> after : ""
	at org.apache.solr.handler.component.QueryComponent.prepare(QueryComponent.java:171)
	at org.apache.solr.handler.component.SearchHandler.handleRequestBody(SearchHandler.java:187)
	at org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:135)
	at net.yacy.cora.federate.solr.connector.EmbeddedSolrConnector.query(EmbeddedSolrConnector.java:179)
	at net.yacy.cora.federate.solr.connector.EmbeddedSolrConnector$DocListSearcher.<init>(EmbeddedSolrConnector.java:345)
	at net.yacy.cora.federate.solr.connector.EmbeddedSolrConnector.getCountByQuery(EmbeddedSolrConnector.java:364)
	at net.yacy.cora.federate.solr.connector.MirrorSolrConnector.getCountByQuery(MirrorSolrConnector.java:326)
	at net.yacy.cora.federate.solr.connector.ConcurrentUpdateSolrConnector.getCountByQuery(ConcurrentUpdateSolrConnector.java:440)
	at net.yacy.search.index.Segment.getWordCountGuess(Segment.java:464)
	at net.yacy.data.DidYouMean.getSuggestions(DidYouMean.java:181)
	at suggest.respond(suggest.java:73)
2014-01-08 04:46:21 +01:00
reger
fbc1071f6d Merge origin/master 2014-01-07 22:48:45 +01:00
reger
7b800a0c8e fix: NPE on shutdown via script 2014-01-07 22:44:24 +01:00
Michael Peter Christen
bfaf42b0b8 added a script which can check the solr index for inconsistencies while
the peer is down. This shall be used in emergency cases where a check or
fix for a broken solr index is needed.
2014-01-07 21:58:55 +01:00
Michael Peter Christen
ce4d42d77c Merge branch 'master' of ssh://git@gitorious.org/yacy/rc1.git 2014-01-07 21:52:38 +01:00
Michael Peter Christen
644573cfc4 using the adminAccountUserName from yacy.conf within apicall.sh 2014-01-07 21:52:19 +01:00
reger
6932aa4d7a use configured admin-username for api calls
- the admin user name can be configured, in apiExec calls the default "admin" username is used. 

TODO: the bin/apicall.sh script should likely take that into account.
2014-01-07 21:26:50 +01:00
reger
c656e67c97 fix: display proper error msg on admin user change 2014-01-07 20:34:37 +01:00
orbiter
2ead4e44d9 introduced a new storage path ARCHIVE inside of DATA which will be used
as path for solr index dumps (instead of the SEGMENTS path). This will
make a maintenance of index backups easier. It will also provide a tool
to migrate from an freeworld index to a webportal index.
2014-01-07 17:53:49 +01:00
sixcooler
add0e42804 fix double-escaped urls from proxy-usage 2014-01-07 01:04:33 +01:00
sixcooler
865ce6f974 check blacklist proxyClient config 2014-01-07 01:01:55 +01:00
sixcooler
345f9aba27 make use of our DNS-cache again - this realy speeds up the lookup 2014-01-07 00:18:01 +01:00
reger
e6d284fe1e better solution for prev. commit with MultiMapSolrParams.getFieldInt not returning default parameter 2014-01-06 18:19:54 +01:00
reger
0bc2fc14ab improve NPE chance on missing parameters
java.lang.NullPointerException
	at net.yacy.http.servlets.SolrServlet.service(SolrServlet.java:145)
	at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:684)
	at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:501)
2014-01-06 17:52:21 +01:00
reger
f06cef5d5b reimplement proxy access by configured whitlist pattern
was currently limited to own ip.
2014-01-06 15:00:14 +01:00
reger
05d6cc6ea3 setting of IPv4Stack moved earlier
it seems even better to call system.setproperty before isrunning check
(if nothing helps we have to set it in startup script)
2014-01-06 11:28:05 +01:00
reger
1b6d173b14 update to Jetty 8.1.14 2014-01-06 08:48:43 +01:00
reger
41dc0f82c1 open service manager upon install failure in installYaCyWindowsService.bat
likely service is already installed
Service Manager allows then to modify settings
2014-01-06 07:22:23 +01:00
reger
30d925a96e reimplemented server access restriction
via Jetty IPAccessHandler to allow only configured IP's to access.
Handler is only loaded if a restriction is configured.

Since IPAcessHandler (Jetty 8) does not support IPv6 system property java.net.preferIPv4Stack=true
Testing showed system.setProperty seems to be sensitive to point of calling (earliest possible time seems to be best = early in yacy.main).
Moved the "isrunning..." just open browser check also to the new routine to preread the yacy.config only once.
2014-01-06 07:00:16 +01:00
orbiter
3cb6c7861f fixed shutdown authenticaton problem 2014-01-06 01:48:54 +01:00
Michael Peter Christen
ed06b5b94b set a realm message to log-in input window which explains that a
password for the account 'admin' can be (re-)set with the script
bin/passwd.sh
2014-01-05 17:43:34 +01:00
Michael Peter Christen
7005ecdabd cleanup 2014-01-05 15:06:40 +01:00
Michael Peter Christen
2939b47986 removed non-working realm setting in http client (auth for localhost was
added in previous commit)
2014-01-05 15:04:18 +01:00
orbiter
9d52b337f3 added http authentification to YaCy http client for all localhost
acesses to enable self-steering of the peer using the API table. This is
necessary in case that an password for the administration pages is set.
2014-01-05 14:46:11 +01:00
Michael Peter Christen
c951945666 modified log-in detail to enable admin-login from localhost with stored
hash even if localhost access is disabled. This is urgently needed for
the apicall.sh script since that is used for high-availability set-up
(checkalive and indexdump for index mirroring)
2014-01-05 11:50:23 +01:00
Michael Peter Christen
9bd71fdbb4 made the access tracker class static because it shall be used by the
jetty auth module
2014-01-05 05:04:28 +01:00
Michael Peter Christen
1c56befb93 fixed mess with test on localhost (which means local hosts for some
cases)
2014-01-05 04:55:30 +01:00
Michael Peter Christen
7d6fc79eb8 refactoring (usage of constant names for attributes of authentication
check)
2014-01-05 04:23:44 +01:00
Michael Peter Christen
b9d36e45e0 removed the &amp explicit encoding of ampersand character since this is
double-translated within the template replacement process.
2014-01-05 03:40:10 +01:00
reger
e2ccb6ce9d modified DefaultServlet parameter on invoke templates
call response with post=0 (if post empty) simulating previous behavior.

(template servlets typically test for post==null,
found one more Crawler.p.java were empty post caused problem,
= defaults not correctly set)
2014-01-04 20:49:26 +01:00
reger
cabe0943cd fix opensearch resultcount in yacysearch.rss
see merge request https://gitorious.org/yacy/rc1/merge_requests/24
use result count in searchtrailer.xml which is on p2p search more accurate (timing)
2014-01-04 17:14:10 +01:00
reger
eaf596a257 adding proxy status to (private) status box
(show also transparent and url proxy status)

show search result via url proxy only if status=on
2014-01-04 16:10:54 +01:00
reger
4c38bceafc handle http connect for proxy
refactor header cleanup (reuse existing code)
2014-01-04 13:09:34 +01:00
reger
cfabe8f67a harmonize access restriction for urlproxy servlet
with proxy handler, what is currently
- use switched on in config
- access from a local IP / hostname

fix shutdown exception for crashprotection handler on interrupted connections.
2014-01-03 12:28:40 +01:00
reger
e3d8459906 extend ssl enabled msg on status page
- post the portnr
2014-01-03 02:56:09 +01:00
reger
e6b9643fd6 extended request for local peer check to by hostname resolved ip
the current islocal() check did not detect a domain.com address as request for the local peer.
2014-01-03 01:13:56 +01:00
reger
c797f108a1 add error response on deniedl proxy access
send http 403 response
2014-01-02 09:11:08 +01:00
reger
0583f44306 reimplement proxy access log (to Jetty ProxyHandler)
- using existing HTTPDProxyHandler logger
- allow local loopback ip to access proxy
2014-01-02 03:37:33 +01:00
reger
8cbc1c970a Security Hot-Fix: for transparent proxy. 2014-01-01 20:48:35 +01:00
reger
58ecf5e4dd add to blacklist button in CrawlResults
http://bugs.yacy.net/view.php?id=220
introduced Blacklist.add with sourcefile only parameter
2014-01-01 11:01:22 +01:00