Commit Graph

162 Commits

Author SHA1 Message Date
luccioman
8da3174867 Ensure lower case conversion consistency with any default locale.
Especially for Turkish speaking users using "tr" as their system default
locale : strings for technical stuff (URLs, tag names, constants...)
must not be lower cased with the default locale, as 'I' doesn't becomes
'i' like in other locales such as "en", but becomes 'ı'.
2017-06-27 06:42:33 +02:00
luccioman
8399275142 Properly close file output streams even on exceptions scenarios. 2017-06-08 07:19:16 +02:00
luccioman
a04feac064 Ensure file input streams proper closing in both success and failures
Also add when possible a warning level log message on input stream
closing error instead of failing silently. This could help understanding
some IO exceptions such as "too many files open".
2017-06-03 04:00:46 +02:00
luccioman
532981b363 Updated putHTML() JavaDoc 2017-05-04 11:21:27 +02:00
reger
3dd23c178b Introduce the option to configure a shutdown port.
A port value of -1 will disable this option.

If set to a value greater 0, YaCy listens on this of on the local loopback 
address (127.0.0.1) for a shutdown or restart signal.
E.g. connect to http://localhost:8005/shutdown will stop the YaCy server.
http://localhost:8005/restart will restart it.
This option allows to stop YaCy locally independant from the web web 
frontend (which might be configured for password protected remote access).
2017-03-19 02:30:08 +01:00
reger
ce87025462 further avoid to set connect info properties as header value
following comment "use of properties as header values is discouraged"
in case where (proxy)HTTPClient overwrites values with supplied url.
Use defined request.referer procedure in response class.
2017-03-04 22:45:17 +01:00
luccioman
e3892b0957 Added some JavaDoc. 2017-01-06 11:23:40 +01:00
reger
b32bcdf344 list entries in outgoing cookie monitor one per line
for easier readability.
For this adjust outgoingCookies entry to use Cookie[] instead of String[]
2016-12-10 22:08:09 +01:00
reger
9a8691129f fix typing error from commit 60ba5c117c 2016-11-25 04:18:36 +01:00
reger
f9328f07e2 completing the usage of CONNECTION_PROP_CLIENT_HTTPSERVLETREQUEST in
HTTPDProxyHandlers logging facility.
2016-11-25 03:05:53 +01:00
reger
8e3e3ed191 update the older ResponseHeader patch to handle cookies,
to work directly with javax.servlet.http.Cookie (rename headerProps to
cookieStore as is only used for this).
(Re)implement set-cookie in DefaultServlet to make cookieAuthentication
work as designed.
2016-11-25 02:00:20 +01:00
reger
4eeb448eb3 use DigestURL in UrlProxyServlet as parameter to pass requested url to
handler.
UrlProxyServlet splits url in parts to pass it on as parameter and 
HeaderFramework constructs a url from param parts. This is obsolete if
already created url is used (makes HeaderFramework.getRequestURL obsolete
= removed)
2016-11-20 05:27:33 +01:00
reger
9db68acb4f remove obsolete X_YACY... header declarations
not in use (no writes, only remove and try to read).
Obsolete parameter setupHttpClient
2016-11-14 22:53:26 +01:00
reger
395f2e8946 Make ServletRequest implement the standardized HttpServletRequest interface,
to make all readily available information from the original ServletRequest
available to YaCy servlets (without converting data to internal structures).
The implementation of the common interface allows easier integration of
YaCy servlets with the servlet standard (e.g. shared login service with
the servlet container etc.)
2016-11-14 01:37:16 +01:00
luccioman
7296e3884f Switched even more URLs to pure relative ones.
Thus a YaCy peer can run behind a reverse proxy subfolder without need
for the reverse proxy to rewrite HTML links (a CPU costly operation).

Tested on Debian Jessie with an apache2 reverse proxy.

See related mantis issues http://mantis.tokeek.de/view.php?id=106 and
http://mantis.tokeek.de/view.php?id=701
2016-11-09 02:40:33 +01:00
luccioman
54d879a9b3 Generate HTML relative (to each peer) links from hosted WikiCode.
When WikiCode inserted in a peer hosted Blog, Wiki, Messages or Profile
contains relative links (images or any content, hosted in DATA/HTDOCS),
it is more reliable to keep these links relative, especially when the
peer is behind any kind of reverse Proxy.
2016-11-04 11:21:20 +01:00
Michael Peter Christen
c716648c78 enhanced json encoding of strings 2016-09-06 18:45:29 +02:00
luccioman
8255e91c99 Fixed serverClassLoader.findClass method
htroot is a supposed to be a subfolder of appPath and not of dataPath,
as assumed in other places where htroot is loaded. This issue was not
visible when dataPath and appPath are equals.
2016-09-03 15:21:02 +02:00
Orbiter
503312ca43 Merge pull request #61 from luccioman/heroku_experiments
Deploy YaCy on Heroku
2016-08-26 11:57:41 +02:00
reger
33bf35d90f missing file for prev commint "Introduction of additional language setting browser" 2016-08-23 00:13:20 +02:00
reger
16e8ed3f01 Introduce additional language setting "browser/Browser Language" for UI internationalization.
If language is set to "browser" the client/user browser language is used to choose from
available translation.
simply: one users browser speaks English -> YaCy responds in English, other users browser speaks French -> YaCy responds in French.

! To make a translation/language available you have to activate the language once ! 
(or manually use the utility class TranslateAll)
In ConfigBasic.html availabel translations are marked green on setting language=Browser
The client language is determined by http header Accept-Language (checked in DefaultServlet)
2016-08-23 00:04:24 +02:00
reger
3b47a07dd1 change unused servletProperties entry CONNECTION_PROP_CLIENT_REQUEST_HEADER to
use directly HttpServletRequest. This is used to get the http protocol version
in HTTPDProxyHandler.fulfillRequestFromWeb() for error response to client.
- adjust YaCyProxyServlet and UrlProxyServlet accordingly
- use more http_version constants in headerframework and httpdeamon
- equalize servlets (3) use of HeaderFramework.CONNECTION_PROP_HOST to HeaderFramework.HOST
2016-08-21 19:34:44 +02:00
luccioman
9b9c112263 Handle more propertly local port configuration by system property
And prefixed property with "net.yacy" to avoid ambiguity.
2016-07-12 01:53:01 +02:00
luccioman
b57a06d88e Let Heroku decide which http port to use 2016-07-06 22:14:40 +02:00
reger
38e2b054d4 remove servlet classloder internal cache map (to save the resources, cache hits marginal)
- DefaultServlet includes already a class cache "templateMethodCache" which is emptied 
  on low mem status 
- avoid classloader cache gets has no hits but over time holds all (used) servlet classes
2016-02-12 01:20:03 +01:00
reger
c28142095a add findClass() to servlet class loader (used in YaCyDefaltServlet)
In the 2 cases where servlet calls servlet the jvm classloader chain is
invoked and servlet class loaded by jvm loader (successful while requiring 
htroot in system classpath). This patch uses the standard override design
for loaders to handle these cases (making in not longer crucial to have htroot 
in system classpath, as this classLoader is mainly used for servlets and
looks in this case for the class in the configured path).
+ As the default classloader is parallelcapable we should register this too.
2016-02-02 03:44:01 +01:00
luc
571bc55937 Refactoring : use StandardCharsets constants instead of hard-coded
charset names.
2016-01-05 23:37:05 +01:00
reger
b7e8358645 make use of header.getContentType where possible (mime is normalized afterwards)
otherwise use header.mime() differentiated in prev. commit.
2015-12-20 15:49:24 +01:00
reger
a60b1fb6c2 differentiate api call getLocalPort() from getConfigInt() 2015-10-31 23:09:03 +01:00
reger
43c27aa550 upd to solr/lucene 5.3.1 2015-10-03 23:20:33 +02:00
reger
206883f80d fix: Preserve protocol in url proxy
to connect to http/https. Display warning if https target is viewed over http
2015-08-25 01:16:41 +02:00
reger
ee77f24e52 use some more declared HeaderFramework constants 2015-08-02 22:56:14 +02:00
Michael Peter Christen
fed26f33a8 enhanced timezone managament for indexed data:
to support the new time parser and search functions in YaCy a high
precision detection of date and time on the day is necessary. That
requires that the time zone of the document content and the time zone of
the user, doing a search, is detected. The time zone of the search
request is done automatically using the browsers time zone offset which
is delivered to the search request automatically and invisible to the
user. The time zone for the content of web pages cannot be detected
automatically and must be an attribute of crawl starts. The advanced
crawl start now provides an input field to set the time zone in minutes
as an offset number. All parsers must get a time zone offset passed, so
this required the change of the parser java api. A lot of other changes
had been made which corrects the wrong handling of dates in YaCy which
was to add a correction based on the time zone of the server. Now no
correction is added and all dates in YaCy are UTC/GMT time zone, a
normalized time zone for all peers.
2015-04-15 13:17:23 +02:00
Michael Peter Christen
a8a2b7a803 persistency for vocabulary facet switch 2015-01-29 02:16:42 +01:00
Michael Peter Christen
efbc9a3561 introducting a new getConfig method which parses comma-separated llists
from setting fields; refactoring for all places where such lists are
parsed
2015-01-29 01:53:36 +01:00
Michael Peter Christen
69eacdf4eb applying precompiled CommonPattern.COMMA.split to all places where
split(",") was used
2015-01-29 01:46:22 +01:00
Michael Peter Christen
5a060c9f26 refactoring of reindexSolr (just replaced constant string) 2015-01-29 00:33:07 +01:00
Michael Peter Christen
bee5ee7cce removed some warnings 2015-01-27 17:00:20 +01:00
arucard21
3e9871291f Applied URL-decoding prior to HTML-encoding.
This removes percent-encoding from text shown in HTML
2014-12-27 09:52:34 +01:00
reger
1f9389396a fix NPE related 500 (Bad Request) response of UrlProxy on blacklisted urls,
by adding parameter HTTPDeamon and removing unused hostAddress lookup code in sendRespondError
2014-12-25 02:21:45 +01:00
reger
f856edecb6 fix proxy redirect (http status 302) response
fixes http://mantis.tokeek.de/view.php?id=517

The url given in bug report uses a gzip input stream which causes the HTTPClient.writeto() throw an IOException due to incomplete input stream. This in turn prevents the 302 reponse to the client browser. 
By limiting to serve target content just on httpstatus=200 will proxy the header reponse and client browsers redirect settings can be honored.
2014-12-23 02:01:03 +01:00
Michael Peter Christen
28683530cd fixes to usage of no-cache: use and recognize also the no-store
directive
2014-12-19 17:37:58 +01:00
reger
70cf7060a4 coding fixes suggested in
http://mantis.tokeek.de/view.php?id=509
http://mantis.tokeek.de/view.php?id=510
2014-12-06 01:42:24 +01:00
reger
ff18129def ViewFile servlet: update index if newer,
so viewed text and metadata (stored) info is similar
- to archive it, use request with profile to allow indexing (defaultglobaltext) and update index 
   (the resource is loaded, parsed anyway, so it's not a expensive operation)

Request: remove 2 unused init parameter 
- number of anchors of the parent
- forkfactor sum of anchors of all ancestors
2014-12-05 01:13:37 +01:00
reger
28456dfc09 skip creation of unused Bluelist contenttransformer 2014-12-02 21:03:00 +01:00
Michael Peter Christen
167c5a51f0 IPv6 fix 2014-10-28 15:36:13 +01:00
Michael Peter Christen
d8beafba3a fix for values in CrawlProfileEditor table and xml; now the full profile
is available in the xml.
2014-10-09 13:27:20 +02:00
Michael Peter Christen
97995a1dd9 fix for remote search process 2014-10-07 23:30:32 +02:00
Marc Nause
1e6e69bc40 Finished implementation of UPNP:
*) will try other ports if YaCy standard ports are not available
*) distinguish between internal and external port (not sure if this
works 100%)

Still to add: propery in config to enter own external port (in case of
manually configured NAT)
2014-10-07 13:10:06 +02:00
Michael Peter Christen
e1bc768f9d more IPv6 bugfixes 2014-10-06 17:44:27 +02:00