Michael Peter Christen
3276508d1b
Merge branch 'master' of ssh://git@gitorious.org/yacy/rc1.git
2012-07-31 23:49:56 +02:00
Michael Peter Christen
3ce04cecf3
bad hack to prevent a bug appearing in solr
2012-07-31 23:49:07 +02:00
sixcooler
f32aa9a49c
prevent merge of blobs that can't be handled in memory
2012-07-31 23:23:16 +02:00
Michael Peter Christen
bbd242afb4
fix for a NPE
2012-07-30 14:51:01 +02:00
Michael Peter Christen
24d9db1613
snippet retrieval loading processes may use a smaller minimum load time
...
value than crawling processes. This speeds up the search result
preparation dramatically.
2012-07-30 10:38:23 +02:00
Michael Peter Christen
ef488a15f7
Merge branch 'master' of ssh://git@gitorious.org/yacy/rc1.git
2012-07-27 12:14:24 +02:00
Michael Peter Christen
1687737771
Abstraction of HandleMap and HandleSet
2012-07-27 12:13:53 +02:00
sixcooler
76b037a20a
check content domain fix:
...
search image/media should not show pages containing image/media
search text should show all/text but image/media
2012-07-27 04:11:52 +02:00
Michael Peter Christen
e432bb9cd9
better calculation of possible saving in HeapReader index data structure
2012-07-26 10:05:06 +02:00
Michael Peter Christen
9549984c65
documentation/comments
2012-07-25 21:34:23 +02:00
Michael Peter Christen
3bcd9d622b
cleaned up classes and methods which are either superfluous at this time
...
or will be superfluous or subject of complete redesign after the
migration to solr. Removing these things now will make the transition to
solr more simple.
2012-07-25 14:31:54 +02:00
Michael Peter Christen
6f1ddb2519
Moved solr index-add method to the same method where the YaCy index is
...
written. Also done some code-cleanup.
2012-07-25 01:53:47 +02:00
Michael Peter Christen
315d83cfa0
cleanup
2012-07-24 22:16:56 +02:00
Michael Peter Christen
1f41d9c6f5
bugfix for a NPE
2012-07-24 17:29:32 +02:00
Michael Peter Christen
76202f068e
extended abstraction of local and remote solr index using one front-end
...
for index administration and querying.
2012-07-24 17:23:29 +02:00
Michael Peter Christen
d3f243e2e1
fixed node type calculation for principal peers
2012-07-23 23:40:50 +02:00
Michael Peter Christen
826967513b
changed options in IndexFederated_p to switch on/off parts of the index
...
individually. The settings are experimental and the values of the
settings will be overwritten when an index migration from urldb to solr
starts.
2012-07-23 16:28:39 +02:00
Michael Peter Christen
cba4ab862e
fix for http://bugs.yacy.net/view.php?id=202
2012-07-23 00:36:18 +02:00
orbiter
69e743d9e3
- more abstraction for the RWI index as preparation for solr integration
...
- added options in search index to switch parts of the index on or off
2012-07-22 13:18:45 +02:00
orbiter
05a3ffd03a
patches to ensure that solr connectors are active ony if they have a
...
solr object assigned and vice versa
2012-07-20 11:47:50 +02:00
orbiter
5a3c829872
embedded solr is only initiated if it is activated with
...
IndexFederated_p.html
2012-07-20 11:40:33 +02:00
Michael Peter Christen
97b7bcf2a6
added a solr search index
...
- by default, a (empty) solr storage instance is created at
SEGMENTS/solr_36
- the index is written if in /IndexFederated_p.html the flag "embedded
solr search index" is switched on
- a standard solr query interface is available now with a new servlet at
http://127.0.0.1:8090/solr/select
To test this, do the following:
- switch to webportal mode
- switch on the feature as described
- do a crawl. this fills the solr index. The normal YaCy search will NOT
work now!
- do a solr query, like:
http://127.0.0.1:8090/solr/select?q= *:*
http://127.0.0.1:8090/solr/select?q=text_t:Help
play with different search fields as you can see in
/IndexFederated_p.html
You can use the standard solr query attributes as described in
http://wiki.apache.org/solr/SearchHandler
2012-07-19 11:34:05 +02:00
Michael Peter Christen
f0a079ac9f
allow larger log entries
2012-07-14 16:28:14 +02:00
Michael Peter Christen
784a4abb18
enhancement in internal data organization which should generate less
...
synchronizations in database access
2012-07-14 13:09:44 +02:00
Michael Peter Christen
f78ce93a80
collection of speed and memory saving hacks
2012-07-13 21:15:38 +02:00
orbiter
c00a3cf74d
less usage of generic logger to avoid logger generation overhead
2012-07-12 19:54:54 +02:00
orbiter
a196f24f60
prevent enqueueing of non-loggeable logging entries
2012-07-12 19:42:42 +02:00
orbiter
482afed07c
reduced logging overhead (a bit)
2012-07-12 19:23:40 +02:00
orbiter
e76159040b
Merge branch 'master' of ssh://git@gitorious.org/yacy/rc1.git
2012-07-12 11:14:04 +02:00
orbiter
bbfa497a3c
replaced more size() > 0 by !isEmpty()
2012-07-12 11:12:21 +02:00
Michael Peter Christen
58e7d1952f
reduction of logging to prevent too much IO caused be logging
2012-07-12 02:08:11 +02:00
Michael Peter Christen
83da68c4c1
fixed a memory leak inside the logger which appeared if the log was
...
writter faster that the logger is able to print this out to its out
stream. A very large collection of unwritten log outputs had been seen
during strong crawling. The new ArrayBlockingQueue is limited to prevent
this case.
2012-07-12 01:23:04 +02:00
orbiter
0cbda0b2b8
- replaced all length() == 0 and size() == 0 with isEmpty()
...
- replaced some length() > 0 and size() > 0 with !isEmpty() - cannot be
done automatically
- implemented some isEmpty() methods
2012-07-10 22:59:03 +02:00
orbiter
28b30231c3
fix for url matcher of multiple amp& in an url, see:
...
http://forum.yacy-websuche.de/viewtopic.php?f=8&t=4439&p=26650#p26650
2012-07-10 17:39:56 +02:00
Roland 'Quix0r' Haeder
aef9dd0350
- removed cleaning of blacklist cache on startup
...
- added cleaning of blacklist cache if cache is modified in interface
- extended cache saving to all cache types
- moved cache location to DATA/LISTS
- fixed static file path which was relative to the application path but
should be relative to data path - which is different in debian and mac
implementations
2012-07-10 13:08:16 +02:00
orbiter
c7afa8bc48
using SwitchboardConstants for solr attributes
2012-07-10 12:01:20 +02:00
orbiter
c6d8950651
Merge branch 'master' of ssh://git@gitorious.org/yacy/rc1.git
2012-07-09 14:33:11 +02:00
orbiter
5f3b8dc040
fix for RSS reader
2012-07-09 14:32:35 +02:00
orbiter
62202e2d71
refactoring of query attribute variable names for better consistency
...
with (next) stored query words
2012-07-09 11:14:50 +02:00
Michael Peter Christen
1addbc792c
use less memory for md5 cache
2012-07-08 22:05:04 +02:00
Michael Peter Christen
f32de94723
more logging
2012-07-08 22:04:36 +02:00
Michael Peter Christen
d09d9f2364
filter old peers from bootstrap (now stronger: 60 minutes instead of
...
240).
2012-07-08 21:25:22 +02:00
Michael Peter Christen
434ee90c59
added classification for control file types which shall not be loaded
...
but placed onto the noload-queue
2012-07-08 21:17:33 +02:00
Michael Peter Christen
a90bcb48f6
added webm
2012-07-08 17:58:05 +02:00
Michael Peter Christen
801972fe6f
fix for url camel case parser and sentence reader
2012-07-08 16:48:09 +02:00
Michael Peter Christen
fbc1a2030d
fix for sitemap importer: can now also import very large sitemaps within
...
small memory configurations
2012-07-08 16:11:50 +02:00
Michael Peter Christen
92731e5287
fix for sevenzip parser
2012-07-08 16:11:19 +02:00
Michael Peter Christen
45641b0c23
catch and log a warning in RasterPlotter
2012-07-06 09:21:12 +02:00
Michael Peter Christen
8efc1c1078
- fixed a memory leak (or bad usage) during parsing/snippet fetch
...
- more logging for errors
2012-07-06 09:05:41 +02:00
Michael Peter Christen
c3db015410
prevent loading of content from the cache when retrieval with IFFRESH is
...
used and cache is stale. Should speed up snippet generation when cache
strategy is IFFRESH.
2012-07-06 08:29:41 +02:00
Michael Peter Christen
b1e7c11fba
fix for pattern matcher in html parser
2012-07-05 14:24:03 +02:00
Michael Peter Christen
8a6edc0031
fix for solr shutdown
2012-07-05 14:23:43 +02:00
Michael Peter Christen
b8bcc06283
fix for urls beginning with "//"
2012-07-05 14:23:29 +02:00
Michael Peter Christen
b0c408788b
made class methods static where possible
2012-07-05 12:38:41 +02:00
Michael Peter Christen
5bd3c90907
- removed unnecessary semicolons
...
- added default case for switch
2012-07-05 11:18:31 +02:00
Michael Peter Christen
132afaf687
removed unaccessible code
2012-07-05 11:09:44 +02:00
Michael Peter Christen
7c1ba99755
removed more unused method parameters
2012-07-05 10:44:30 +02:00
Michael Peter Christen
83701a1b4c
removed unused ImageReference package
2012-07-05 10:24:52 +02:00
Michael Peter Christen
0301aba1e9
removed unused method parameters
2012-07-05 10:23:07 +02:00
Michael Peter Christen
241dd8410a
removed snippet pattern filter - it was not used
2012-07-05 09:21:27 +02:00
Michael Peter Christen
d3964253ae
- added @SuppressWarnings to unused servlet method parameters
...
- removed unnecessary casts
- removed unnecessary throw statements
2012-07-05 09:14:04 +02:00
Michael Peter Christen
ea10766bfd
cleaned unnecessary nested code
2012-07-05 08:44:39 +02:00
Michael Peter Christen
1481037820
replaced non-generic array with collection
2012-07-05 01:02:51 +02:00
orbiter
fc0f9543fe
More SentenceReader cleanup
2012-07-05 00:20:58 +02:00
orbiter
586bb0eb6a
Simplified SentenceReader (no more Reader inside..)
2012-07-04 22:06:20 +02:00
orbiter
7f851d62a7
replaced HashARC with SizeLimited Objects which are less costly
2012-07-04 21:56:25 +02:00
orbiter
d4291ac1f3
more tolerance when creating solar document
2012-07-04 21:15:38 +02:00
orbiter
78fc3cf8f8
refactoring and new usage of SentenceReader: this class appeared as one
...
of the major CPU users during snippet verification. The class was not
efficient for two reasons:
- it used a too complex input stream; generated from sources and UTF8
byte-conversions. The BufferedReader applied a strong overhead.
- to feed data into the SentenceReader, multiple toString/getBytes had
been applied until a buffered Reader from an input stream was possible.
These superfluous conversions had been removed.
- the best source for the Sentence Reader is a String. Therefore the
production of Strings had been forced inside the Document class.
2012-07-04 21:15:10 +02:00
orbiter
bb8dcb4911
automatically adopt size of word cache to available memory
2012-07-03 18:22:25 +02:00
Michael Peter Christen
ad09b786bf
clean up parser data
2012-07-03 17:20:41 +02:00
Michael Peter Christen
276a66a793
Adding a limit of 1000 links that a parser shall store during indexing.
...
A limit was necessary because some web pages have such huge numbers of
links that it can easily cause a OOM just by the number of links.
The quesion if the number of 1000 links is sufficient or too weak must
be answered with the result of testing this feature.
2012-07-03 17:06:20 +02:00
Michael Peter Christen
613b45f604
- better data structures in secondary search
...
- fixed a big memory leak in secondary search
2012-07-03 07:12:20 +02:00
Michael Peter Christen
de903a53a0
parser refactoring & hacks
2012-07-03 06:06:38 +02:00
Michael Peter Christen
8a82609360
- smaller caches to save memory
...
- close cloneable iterators to free memory
2012-07-02 15:40:40 +02:00
Michael Peter Christen
7249d9c9de
bugfix for concurrent seed loader
2012-07-02 14:37:57 +02:00
Michael Peter Christen
c72d3b12cd
concurrently initialize the seed list during p2p network bootstrap
2012-07-02 14:27:37 +02:00
Michael Peter Christen
1825f165b8
better integration of blacklist according to use case
2012-07-02 13:57:29 +02:00
Michael Peter Christen
c18fa9fa75
Merge branch 'master' of git://gitorious.org/~reger/yacy/bbyacy-rc1
2012-07-02 12:20:57 +02:00
Michael Peter Christen
ce8d4b87d9
fixes for new eclipse 'Juno' warning 'Resource leak'.
2012-07-02 10:27:46 +02:00
Michael Peter Christen
0c345d1559
giving threads name so its easier to see whats happening during
...
debugging and within a thread dump
2012-07-02 09:51:43 +02:00
reger
067728bccc
add search result heuristic. adding a crawl job with depth-1 for every displayed search result (crawling every external linked page of displayed search result pages)
2012-07-01 00:12:20 +02:00
Michael Peter Christen
03280fb161
removed segments-concept and the Segments class:
...
the segments had been there to create a tenant-infrastructure but were
never be used since that was all much too complex. There will be a
replacement using a solr navigation using a segment field in the search
index.
2012-06-28 14:27:29 +02:00
Michael Peter Christen
508a81b86c
added solr field 'refresh_s' which stores the refresh url contained in
...
the meta-refresh html header field.
2012-06-28 13:27:45 +02:00
Michael Peter Christen
f3167def64
do not fill the keywords with title content if keywords do not exist.
2012-06-27 13:07:02 +02:00
Michael Peter Christen
9116013c64
- allow lazy initialization of solr value (if using 'lazy', then no
...
0-values and no empty strings are written). This may save a lot of
memory (in ram and on disc) if excessive 0-values or empty strings
appear)
- do not allow default boolean values for checkboxes because that does
not make sense: browsers may omit the checkbox attribute name if the box
is not checked. A default value 'true' would not comply with the
semantic of the browsers response.
- add a checkbox in IndexFederated_p for the lazy initialization of solr
fields.
2012-06-27 12:17:58 +02:00
sixcooler
97f60010d8
fix crawl start from file
2012-06-26 16:11:39 +02:00
Michael Peter Christen
0294a53459
- add canonical field only if requested by solr schema
...
- remove canonical url from in/outbound urls if present
2012-06-26 14:51:57 +02:00
Michael Peter Christen
3fd4a01286
added option to record urls that are forwarded to the solr index
2012-06-26 13:54:48 +02:00
Michael Peter Christen
d763e4d94b
fixed bad referer computation in SSIs which causes a NPE during host
...
computation. This error was there before the latest IPv6 hack but did
not cause a NPE. The IPv6 hack was not the cause for this bug, but it
discovered the misconfiguration of the 'referer' referrer.
2012-06-26 11:18:29 +02:00
Michael Peter Christen
358b04885e
more IPv6 hacks
2012-06-26 00:25:46 +02:00
Michael Peter Christen
96aeb127e3
generalized localhost naming.
...
this is also a preparation for a better IPv6 implementation.
2012-06-26 00:08:25 +02:00
Michael Peter Christen
77f795756c
fixing redirects and status codes: storing of status code in
...
ResponseHeader to make it available for late evaluations, like storage
in solr.
2012-06-25 18:17:31 +02:00
Michael Peter Christen
8dd469b9dd
added option to configure the autocommit delay time of solr on-the-fly
2012-06-25 14:59:46 +02:00
Michael Peter Christen
b9dfca4b0a
- fixed IndexFederated Servlet / a embedded Solr can now be selected
...
- added code stub for an embedded Solr but generation of Solr store is
still commented out (it works but is not yet ready for usage)
2012-06-25 11:34:38 +02:00
Michael Peter Christen
fad3b14813
added jetty libraries, needed for future use as web server and as
...
application server for the solr search interface
2012-06-22 15:31:17 +02:00
Michael Peter Christen
a38b0a2c46
extended embedded solr tests to ensure that it will be usable within a
...
jetty instance
2012-06-22 11:40:02 +02:00
Michael Peter Christen
b9d42fd9c8
using com.google.common.io.Files instead of homebrew methods
2012-06-22 11:39:17 +02:00
Michael Peter Christen
a5eb91fa60
refactoring
2012-06-22 00:49:32 +02:00
Michael Peter Christen
1be0025a9c
- added test for EmbeddedSolrConnector
...
- added needed libraries for this test
this includes most (all) files needed for an embedded solr
2012-06-22 00:36:49 +02:00
Michael Peter Christen
dbdd697f4d
moved RDFaParser.xsl configuration file to defaults
2012-06-21 16:09:12 +02:00