Michael Peter Christen
241dd8410a
removed snippet pattern filter - it was not used
2012-07-05 09:21:27 +02:00
Michael Peter Christen
d3964253ae
- added @SuppressWarnings to unused servlet method parameters
...
- removed unnecessary casts
- removed unnecessary throw statements
2012-07-05 09:14:04 +02:00
Michael Peter Christen
ea10766bfd
cleaned unnecessary nested code
2012-07-05 08:44:39 +02:00
Michael Peter Christen
1481037820
replaced non-generic array with collection
2012-07-05 01:02:51 +02:00
Michael Peter Christen
4de50fe808
adding more principal peers for bootstraping
2012-07-05 00:43:41 +02:00
orbiter
fc0f9543fe
More SentenceReader cleanup
2012-07-05 00:20:58 +02:00
orbiter
586bb0eb6a
Simplified SentenceReader (no more Reader inside..)
2012-07-04 22:06:20 +02:00
orbiter
7f851d62a7
replaced HashARC with SizeLimited Objects which are less costly
2012-07-04 21:56:25 +02:00
orbiter
d4291ac1f3
more tolerance when creating solar document
2012-07-04 21:15:38 +02:00
orbiter
78fc3cf8f8
refactoring and new usage of SentenceReader: this class appeared as one
...
of the major CPU users during snippet verification. The class was not
efficient for two reasons:
- it used a too complex input stream; generated from sources and UTF8
byte-conversions. The BufferedReader applied a strong overhead.
- to feed data into the SentenceReader, multiple toString/getBytes had
been applied until a buffered Reader from an input stream was possible.
These superfluous conversions had been removed.
- the best source for the Sentence Reader is a String. Therefore the
production of Strings had been forced inside the Document class.
2012-07-04 21:15:10 +02:00
orbiter
bb8dcb4911
automatically adopt size of word cache to available memory
2012-07-03 18:22:25 +02:00
Michael Peter Christen
ad09b786bf
clean up parser data
2012-07-03 17:20:41 +02:00
Michael Peter Christen
276a66a793
Adding a limit of 1000 links that a parser shall store during indexing.
...
A limit was necessary because some web pages have such huge numbers of
links that it can easily cause a OOM just by the number of links.
The quesion if the number of 1000 links is sufficient or too weak must
be answered with the result of testing this feature.
2012-07-03 17:06:20 +02:00
Michael Peter Christen
613b45f604
- better data structures in secondary search
...
- fixed a big memory leak in secondary search
2012-07-03 07:12:20 +02:00
Michael Peter Christen
de903a53a0
parser refactoring & hacks
2012-07-03 06:06:38 +02:00
Michael Peter Christen
8a82609360
- smaller caches to save memory
...
- close cloneable iterators to free memory
2012-07-02 15:40:40 +02:00
Michael Peter Christen
7249d9c9de
bugfix for concurrent seed loader
2012-07-02 14:37:57 +02:00
Michael Peter Christen
c72d3b12cd
concurrently initialize the seed list during p2p network bootstrap
2012-07-02 14:27:37 +02:00
Michael Peter Christen
1825f165b8
better integration of blacklist according to use case
2012-07-02 13:57:29 +02:00
Michael Peter Christen
c18fa9fa75
Merge branch 'master' of git://gitorious.org/~reger/yacy/bbyacy-rc1
2012-07-02 12:20:57 +02:00
Michael Peter Christen
ce8d4b87d9
fixes for new eclipse 'Juno' warning 'Resource leak'.
2012-07-02 10:27:46 +02:00
Michael Peter Christen
0c345d1559
giving threads name so its easier to see whats happening during
...
debugging and within a thread dump
2012-07-02 09:51:43 +02:00
reger
067728bccc
add search result heuristic. adding a crawl job with depth-1 for every displayed search result (crawling every external linked page of displayed search result pages)
2012-07-01 00:12:20 +02:00
Michael Peter Christen
03280fb161
removed segments-concept and the Segments class:
...
the segments had been there to create a tenant-infrastructure but were
never be used since that was all much too complex. There will be a
replacement using a solr navigation using a segment field in the search
index.
2012-06-28 14:27:29 +02:00
Michael Peter Christen
508a81b86c
added solr field 'refresh_s' which stores the refresh url contained in
...
the meta-refresh html header field.
2012-06-28 13:27:45 +02:00
Michael Peter Christen
f3167def64
do not fill the keywords with title content if keywords do not exist.
2012-06-27 13:07:02 +02:00
Michael Peter Christen
9116013c64
- allow lazy initialization of solr value (if using 'lazy', then no
...
0-values and no empty strings are written). This may save a lot of
memory (in ram and on disc) if excessive 0-values or empty strings
appear)
- do not allow default boolean values for checkboxes because that does
not make sense: browsers may omit the checkbox attribute name if the box
is not checked. A default value 'true' would not comply with the
semantic of the browsers response.
- add a checkbox in IndexFederated_p for the lazy initialization of solr
fields.
2012-06-27 12:17:58 +02:00
sixcooler
97f60010d8
fix crawl start from file
2012-06-26 16:11:39 +02:00
Michael Peter Christen
c03d306afa
shorter autocommit time (now: 1 second) to prevent that user cannot see
...
results in solr the first time they try it out. The value can now be
easily set to a higher number using the IndexFederated_p interface.
2012-06-26 14:53:45 +02:00
Michael Peter Christen
0294a53459
- add canonical field only if requested by solr schema
...
- remove canonical url from in/outbound urls if present
2012-06-26 14:51:57 +02:00
Michael Peter Christen
3fd4a01286
added option to record urls that are forwarded to the solr index
2012-06-26 13:54:48 +02:00
Michael Peter Christen
d763e4d94b
fixed bad referer computation in SSIs which causes a NPE during host
...
computation. This error was there before the latest IPv6 hack but did
not cause a NPE. The IPv6 hack was not the cause for this bug, but it
discovered the misconfiguration of the 'referer' referrer.
2012-06-26 11:18:29 +02:00
Michael Peter Christen
358b04885e
more IPv6 hacks
2012-06-26 00:25:46 +02:00
Michael Peter Christen
96aeb127e3
generalized localhost naming.
...
this is also a preparation for a better IPv6 implementation.
2012-06-26 00:08:25 +02:00
Michael Peter Christen
77f795756c
fixing redirects and status codes: storing of status code in
...
ResponseHeader to make it available for late evaluations, like storage
in solr.
2012-06-25 18:17:31 +02:00
Michael Peter Christen
8dd469b9dd
added option to configure the autocommit delay time of solr on-the-fly
2012-06-25 14:59:46 +02:00
Michael Peter Christen
5d9bd4ddc2
Merge remote-tracking branch 'origin/master'
2012-06-25 11:37:32 +02:00
Michael Peter Christen
b9dfca4b0a
- fixed IndexFederated Servlet / a embedded Solr can now be selected
...
- added code stub for an embedded Solr but generation of Solr store is
still commented out (it works but is not yet ready for usage)
2012-06-25 11:34:38 +02:00
reger
2931726386
adjusted NetBeans classpath for new and updated libraries in lib
2012-06-24 22:50:08 +02:00
Michael Peter Christen
cc1b6762bb
root, not yacy
2012-06-24 10:58:09 +02:00
Michael Peter Christen
2589158f44
changed recommended line in /etc/crontab for high-availability
2012-06-24 10:57:18 +02:00
reger
4156d4e12b
Merge branch 'master' of git://gitorious.org/yacy/rc1.git
2012-06-23 21:22:46 +02:00
Michael Peter Christen
7b53be141f
upgraded to pdfbox 1.7.0
...
changes in http://www.apache.org/dist/pdfbox/1.7.0/RELEASE-NOTES.txt
with many bugfixes, including performance related
2012-06-22 16:49:58 +02:00
Michael Peter Christen
fad3b14813
added jetty libraries, needed for future use as web server and as
...
application server for the solr search interface
2012-06-22 15:31:17 +02:00
Michael Peter Christen
a38b0a2c46
extended embedded solr tests to ensure that it will be usable within a
...
jetty instance
2012-06-22 11:40:02 +02:00
Michael Peter Christen
b9d42fd9c8
using com.google.common.io.Files instead of homebrew methods
2012-06-22 11:39:17 +02:00
Michael Peter Christen
a5eb91fa60
refactoring
2012-06-22 00:49:32 +02:00
Michael Peter Christen
1be0025a9c
- added test for EmbeddedSolrConnector
...
- added needed libraries for this test
this includes most (all) files needed for an embedded solr
2012-06-22 00:36:49 +02:00
Michael Peter Christen
dbdd697f4d
moved RDFaParser.xsl configuration file to defaults
2012-06-21 16:09:12 +02:00
Michael Peter Christen
90b82ce994
using guava for host resolution (non-blocking for ips) and time-out
2012-06-21 16:04:48 +02:00
