Commit Graph

11573 Commits

Author SHA1 Message Date
Michael Peter Christen
69eacdf4eb applying precompiled CommonPattern.COMMA.split to all places where
split(",") was used
2015-01-29 01:46:22 +01:00
Michael Peter Christen
ac19690d30 refactoring with CommonPattern.COMMA 2015-01-29 01:35:28 +01:00
Michael Peter Christen
cf9b22ca5c do not reindex based on vocabulary fields (there are meanwhile many of
them) and some default settings
2015-01-29 01:22:28 +01:00
Michael Peter Christen
5a060c9f26 refactoring of reindexSolr (just replaced constant string) 2015-01-29 00:33:07 +01:00
Michael Peter Christen
b5a55c8b3d fix for wkhtmltopdf (custom header does not work) 2015-01-28 17:45:25 +01:00
Michael Peter Christen
3d717b749a fix for urlmaskfilter 2015-01-28 13:40:41 +01:00
Michael Peter Christen
2636582435 Merge branch 'master' of ssh://git@gitorious.org/yacy/rc1.git 2015-01-28 10:32:17 +01:00
reger
0260d3d800 Allow to hide linkstructure graphic in crawl monitor
using/setting the config param DECORATION_GRAFICS_LINKSTRUCTURE
2015-01-28 03:59:01 +01:00
Michael Peter Christen
bee5ee7cce removed some warnings 2015-01-27 17:00:20 +01:00
Michael Peter Christen
783cf6fbc7 the LinkedBlockingQueue is much faster than the ArrayBlockingQueue
(strange but this is the result of a test:
ArrayBlockingQueue: 39461 lines / second;
LinkedBlockingQueue: 60774 lines / second)
2015-01-27 16:53:09 +01:00
Michael Peter Christen
6390454652 fix for vocabulary on/off setting 2015-01-27 16:24:27 +01:00
Michael Peter Christen
a3c5995bde Merge branch 'master' of ssh://git@gitorious.org/yacy/rc1.git 2015-01-26 14:13:17 +01:00
reger
5ca0762179 fix: eom on parsing ico file by genericImageParser
trace: java.lang.OutOfMemoryError: Java heap space
	at java.awt.image.DataBufferInt.<init>(DataBufferInt.java:75)
	at java.awt.image.Raster.createPackedRaster(Raster.java:467)
	at java.awt.image.DirectColorModel.createCompatibleWritableRaster(DirectColorModel.java:1032)
	at java.awt.image.BufferedImage.<init>(BufferedImage.java:331)
	at net.yacy.document.parser.images.bmpParser$IMAGEMAP.<init>(bmpParser.java:149)
	at net.yacy.document.parser.images.bmpParser.parse(bmpParser.java:69)
	at net.yacy.document.parser.images.genericImageParser.parse(genericImageParser.java:116)
2015-01-24 23:17:07 +01:00
Michael Peter Christen
4cd2d68e03 Merge branch 'master' of ssh://git@gitorious.org/yacy/rc1.git 2015-01-24 07:10:47 +01:00
Michael Peter Christen
dc5700148f update to latest code changes from json.org 2015-01-24 07:10:14 +01:00
reger
42b0672be3 Let auto-disabled crawls recover if low resource condition vanished.
Analog to autodisabled DHT switch autodisabled crawls back on upon mem ok
by remembering the autodisable by conf parameter.
2015-01-24 01:53:58 +01:00
Michael Peter Christen
b32e0b5457 fix for shell script 2015-01-23 18:34:38 +01:00
Michael Peter Christen
29f6e9db7a write java version to status page 2015-01-23 17:57:54 +01:00
Michael Peter Christen
604ccd8072 new development cycle 2015-01-23 11:31:05 +01:00
Michael Peter Christen
287c528f46 replaced old JavaApplicationStub for Mac Application framework with new
script. Adopted the YaCyApp environment and fixed a problem in the
startYACY.sh application wrapper which caused wrong usage of logging
option -l which caused that files had been written to the YaCy
application folder.
As a result of this fix, it is not necessary any more to change path
settings in Info.plist if libraries are changed.
2015-01-23 11:30:13 +01:00
Michael Peter Christen
2bc2564668 Release 1.82 2015-01-21 12:45:55 +01:00
Michael Peter Christen
4c9d2a7c64 reverted 'do not show all options' strategy. This is actually confusing
new users. Will be activated maybe again if there is an optional
tutorial mode which can be switched on for this special purpose of
running a tutorial.
2015-01-20 18:18:12 +01:00
Michael Peter Christen
7db2888336 fixed font size and print page generation in pdf snapshots 2015-01-20 17:14:14 +01:00
reger
24f68a4eb7 refactor opensearch heuristic
introduce FederateSearchManager handling search heuristic to external systems via specific FederateSearchConnectors,
which provide the query() functionallity, the translation to YaCy schema .toYaCySchema() and the search() routine to deliver results to searchevents, which is generally implemented in Abstract connector.
The manager enforces now a min 15s delay between calls to external systems.
Besides the OpensearchConnector a SolrFederateSearchConnector is available. It uses a additional config file for fieldname translation.

default heuristicopensearch.conf: 
- openbdb.com removed - seems not longer to deliver results
- config via solrconnector to  datacite.org added (large technical library archive)
2015-01-19 03:30:35 +01:00
Michael Peter Christen
3b51636ecb fix for mediawiki import 2015-01-12 00:35:47 +01:00
Michael Peter Christen
b07afbc115 a test with http://validator.w3.org/feed/#validate_by_input shows that
the time format was wrong; we must use RFC-822
2015-01-09 16:45:43 +01:00
Michael Peter Christen
8cafdb989a Merge branch 'master' of ssh://git@gitorious.org/yacy/rc1.git 2015-01-09 11:00:02 +01:00
reger
66839f73fa remove debug limit from commit before 2015-01-09 02:52:18 +01:00
reger
4214f250d0 Add option for extended search (Autosearch) to Bookmark.html asking all connected peers for the searchterm added as description to the bookmark created by the bookmark icon.
Intended for searches/research projects with not sufficient results from local and DHT selected remote target peers.

Function: the process checks newly created bookmarks for description starting with "query=..." and takes this to ask every peer for 20 search results and adds it to the local index in a background job.
link to start/stop the process added to /Bookmarks.html
2015-01-09 02:06:30 +01:00
reger
bb37cb32e4 Add title import for bookmark icon
if avail in index
2015-01-09 01:33:45 +01:00
reger
8e751d754a - add javadoc to busythread with hint about the init parameter useage
- remove obsolete 10_httpd config parameter
2015-01-09 01:31:57 +01:00
Michael Peter Christen
3e6c3e2237 documents pushed over the api/push_p.html interface will have their
unique flag set by default
2015-01-06 15:22:59 +01:00
Michael Peter Christen
0871e43fcc better scale 2015-01-06 14:22:43 +01:00
Michael Peter Christen
35c24608cc fix for division by zero (rare cases) 2015-01-06 14:21:20 +01:00
Michael Peter Christen
4144c7cc52 do not write frame links to webgraph 2015-01-06 14:14:25 +01:00
reger
4eb89d7f15 revert clickservlet
(default was indeed a mistakenly)
2015-01-05 09:10:20 +01:00
Michael Peter Christen
61ae9d2d11 do not use the clickservlet by default. From my personal view, this
technique should not be used at all! This project is about privacy, the
existence of a click servlet is one example why people should NOT use a
search portal if such exists.
2015-01-05 08:21:51 +01:00
Michael Peter Christen
c9e2128260 please commit new files under your own name, this file was not created
by me.
2015-01-05 08:18:19 +01:00
reger
ebe5faeb01 added url to bookmark icon link
url is anyway needed, saves index lookup and works w/o commited url.
Removed unused order parameter
2015-01-05 06:55:53 +01:00
sixcooler
5594c43d2e bump to Solr-/Lucene-4.10.3 2015-01-04 18:47:47 +01:00
reger
d44d8996d0 Added a “don't store remote search results” option
This is intended for peers who want to participate in the P2P network but don't wish to load/fill-up their index with metadata of every received search result. 
The DHT transfer is not effected by this option (and will work as usual, so that a peer disabling the new store to index switch still receives and holds the metadata according to DHT rules).
Downside for the local peer is that search speed will not improve if search terms are only avail. remote or by quick hits in local index.

To be able to improve the local index a Click-Servlet option was added additionally.
If switched on, all search result links point to this servlet, which forwards the users browser (by html header) to the desired page and feeds the page to the fulltext-index.
The servlet accepts a parameter defining the action to perform (see defaults/web.xml, index, crawl, crawllinks)

The option check-boxes are placed in ConfigPortal.html
2015-01-04 11:10:45 +01:00
reger
d729386787 fix NPE in viewimage
Caused by: java.lang.NullPointerException
	at net.yacy.peers.graphics.EncodedImage.<init>(EncodedImage.java:73)
	at ViewImage.respond(ViewImage.java:156)
2015-01-04 09:12:30 +01:00
reger
4ff018c9e4 fix ConfigPortal jumps to iframe focus
add focus parameter to yacysearch.html too
2015-01-04 06:57:13 +01:00
reger
c156548efe add info text to metadata page (htmlresponsewriter) on no documents found 2015-01-04 02:59:21 +01:00
reger
3ac1d14a21 improve TexParser.mimeOf( fileextension ) by returning 1st defined in supported list.
This prevents unusual mapping of supported fileextension -> mimetype
(like htm=application/x-tex)
2015-01-02 04:20:02 +01:00
Michael Peter Christen
d2792a43fd do not write iframe and embed links into webgraph, but use them anyway
for crawling
2015-01-02 02:44:03 +01:00
Michael Peter Christen
5b810f6d70 Merge branch 'master' of gitorious.org:yacy/whitrs-rc1 2015-01-02 00:57:37 +01:00
Ryszard Goń
3cdbd5f5c6 Fix for progress table background not resizing
when the post-processing started/ended.
2015-01-02 00:11:32 +01:00
reger
0dfeee154a adjustments for Bookmark icon to act on BookmarkDB,
it acts on YMarks but YMark interface seems not maintained,
for future features (e.g. query memory) BookmarkDB is the likely choice to expand, besides the crawlstart bookmark also the result bookmark icon now adds to BookmarkDB.
The YMark related code is (for now) left untouched so both tables are updated.
2015-01-01 02:41:20 +01:00
Michael Peter Christen
513e9259f5 Merge branch 'master' of git@gitorious.org:yacy/rc1.git 2014-12-30 02:36:17 +01:00