Michael Peter Christen
8fc3679c66
using more pre-compile pattern for split methods
2012-11-26 13:11:55 +01:00
Michael Peter Christen
d48e9788d2
enhanced search result processing behavior
...
- query less at one time; query more often
- in between the small queries, evaluate results
- remove fields from search results which are not needed
2012-11-26 12:24:35 +01:00
reger
469efcdb9d
fix: display and calculate authors and namespace search navigator if configured (otherwise skip overhead)
...
(leave hosts, topics and not in ConfigPortal included filetype, protocoll navigator untouched)
2012-11-25 22:49:26 +01:00
orbiter
ee612e8b93
start the local search only if this peer is doing a remote search or
...
when it is doing a local search and the peer is old
2012-11-25 11:58:57 +01:00
Michael Peter Christen
d6b82840f8
added a feature to find similarities in documents.
...
This uses an enhanced version of the Nutch/Solr TextProfileSignatue.
As a result, a signature of the document is written to the solr search
index. Additionally for each time when a signature is written, it is
checked if the singature exists already in the index. If the signature
does not exist, the document is marked as unique. The unique attribute
can now be used to sort document lists and bring duplicates to the end
of a result list.
To enable this, a large portion of the search api to Solr had to be
changed. This affected mainly caching of 'exists' searches to enhance
the check for existing signatures and do this without actually doing a
solr query.
Because here the first time a long number is used as value in the Solr
store, also the value naming in the YaCySchema had to be adopted and
normalized. This caused that many files had to be changed.
2012-11-21 18:46:49 +01:00
Michael Peter Christen
46be4af5b9
Merge commit '2bb8f045cc92f31fc7e720cc30b38af417563890'
2012-11-18 22:11:04 +01:00
orbiter
5dfd6359cb
redesign of the QueryParams class: introduced QueryGoal which holds the
...
query string parser. This shall be used to create a proper full-string
matching which is handled then by QueryGoal.
2012-11-18 01:22:41 +01:00
cominch
d2a94cc55e
refactor package
2012-11-09 16:22:24 +01:00
cominch
21df1ad9e0
update and generalization of the SMW import and content control routines
2012-11-09 13:48:40 +01:00
Michael Peter Christen
842faf96a2
fixed media search
2012-11-07 17:27:13 +01:00
Michael Peter Christen
93001586a0
removed warnings, removed too-fast pausing of crawls
2012-11-07 15:37:14 +01:00
Michael Peter Christen
570e42c4e3
fix for filetype naviagtor
2012-11-07 13:53:29 +01:00
Michael Peter Christen
158732af37
automatically delete entries from the crawl profile list if crawl is
...
terminated.
2012-11-07 02:03:44 +01:00
Michael Peter Christen
2371ef031c
added solr faceted search support to YaCy search results
...
added solr highlighting / YaCy snippets to YaCy search results
- facets are now much more complete
- facets are computed and searched much faster
- snippet computation is done by solr if solr knows the snippet
2012-11-06 14:32:08 +01:00
Michael Peter Christen
619bf7e875
fixed filetype modified for media types in text search
2012-11-05 18:08:00 +01:00
Michael Peter Christen
8fb370d9f8
renovated the way how search results are count. should be correct now...
2012-11-05 03:19:28 +01:00
Michael Peter Christen
b764de424a
code cleanup
2012-11-02 10:28:32 +01:00
Michael Peter Christen
1168d09de8
more refactoring - integrated the code of SnippetProcess into
...
SearchEvent
2012-11-01 17:40:06 +01:00
Michael Peter Christen
6629e37685
tried to clean up the search process mess
2012-11-01 17:16:43 +01:00
Michael Peter Christen
c5f67a5d6d
fixed a problem with local search from solr results: now all results
...
from solr are shown (again)
2012-11-01 10:22:22 +01:00
orbiter
276dd6452b
removed warnings
2012-10-23 19:08:44 +02:00
Michael Peter Christen
ce0e5b1e17
- more refactoring / private methods
...
- fix for usage of custom solr field names
2012-10-18 15:09:04 +02:00
Michael Peter Christen
36c13ed15b
less solr prefetch
2012-10-11 10:17:05 +02:00
Michael Peter Christen
584663ae8c
- redesign of solr query construction
...
- fix for solr boosts and location search
- fix for number of search results in local search
2012-10-07 07:46:55 +02:00
orbiter
4fed4a86d8
another fix to location search
2012-10-04 22:44:44 +02:00
Michael Peter Christen
1533bfd63b
refactoring
2012-09-25 21:20:03 +02:00
Michael Peter Christen
e57bf2ca39
simplified DHT classes
2012-09-24 01:04:39 +02:00
Michael Peter Christen
8219a445f3
refactoring
2012-09-21 16:46:57 +02:00
Michael Peter Christen
00c1c777fa
refactoring
2012-09-21 15:48:16 +02:00
Michael Peter Christen
4d29f59a27
removed warnings
2012-09-10 07:15:52 +02:00
Michael Peter Christen
e8acd542b5
- added faceted drill-down for host and geolocation to solr queries
...
- added a new geolocation field to index schema, the old values are
migrated if possible
2012-08-27 14:41:33 +02:00
Michael Peter Christen
ff3eaa21b0
added remote search to solr on YaCy peers!
...
- when doing a remote search, node peers are selected for solr queries
- the solr query is done concurrently to the standard YaCy rwi search
- the solr search result is feeded into the same data structure that
prepares the rwi search result
- the same remote seach that is done to several outside peers is done to
the local solr index
- the search process works now also without any 'old' RWI data using
solr
2012-08-20 12:16:11 +02:00
Michael Peter Christen
a06123aec6
more abstraction and less parameter overhead for remote search
2012-08-20 01:29:15 +02:00
Michael Peter Christen
f00733186b
code simplifications
2012-08-19 13:17:03 +02:00
orbiter
404b0aab09
refactoring in remote search and stub for remote node peer selection
2012-08-18 23:59:25 +02:00
orbiter
0cbda0b2b8
- replaced all length() == 0 and size() == 0 with isEmpty()
...
- replaced some length() > 0 and size() > 0 with !isEmpty() - cannot be
done automatically
- implemented some isEmpty() methods
2012-07-10 22:59:03 +02:00
orbiter
62202e2d71
refactoring of query attribute variable names for better consistency
...
with (next) stored query words
2012-07-09 11:14:50 +02:00
Michael Peter Christen
241dd8410a
removed snippet pattern filter - it was not used
2012-07-05 09:21:27 +02:00
Michael Peter Christen
613b45f604
- better data structures in secondary search
...
- fixed a big memory leak in secondary search
2012-07-03 07:12:20 +02:00
Michael Peter Christen
0c345d1559
giving threads name so its easier to see whats happening during
...
debugging and within a thread dump
2012-07-02 09:51:43 +02:00
Michael Peter Christen
ab7107b34b
fixed RWIProcess queue limits: now discovering hidden results for mass
...
result retrieval
2012-06-08 09:14:54 +02:00
Michael Peter Christen
a1fe65b115
performance hacks
2012-06-05 12:06:26 +02:00
Michael Peter Christen
89142d1e8d
removed (not all) warnings
2012-05-16 13:42:32 +02:00
Michael Peter Christen
ba6aaabc51
refactoring + parser bugfixes
2012-05-04 17:28:27 +02:00
Michael Peter Christen
f8cd57c92f
new indexing strategy: ALL links that appear anywhere are indexed, not
...
only links where the content can be parsed. All non-parseable links are
placed into the noload queue. The search process must therefore be able
to filter out non-text search results.
- This fixes the problem that image search results appeared in the text
search.
- The interactive search can retrieve now ALL types of links
- The p2p interface is now extended to retrieve only certain types of
links (text, image, video, apps)
- The search process has an extension to filter the right document type
according to the search query
2012-04-22 02:05:17 +02:00
Michael Peter Christen
83009d86f7
added the vocabulary navigator. It can be very simply tested by
...
switching on the locale dictionaries.
2012-01-17 01:53:08 +01:00
Michael Peter Christen
c602eaaf46
enhanced search process
2012-01-10 03:00:55 +01:00
Michael Christen
eff966f396
fix for search process (it was aborted too early during remote search)
2012-01-09 03:02:35 +01:00
Michael Christen
0797b0de99
new handling of remote search processes: looking for seeds will now not
...
block the whole search process any more. A deadlock with a DHT selection
process may have been the cause for interface lockings in the past.
2011-12-21 00:32:03 +01:00
Michael Christen
c04bfaa51b
refactoring
2011-12-16 23:59:29 +01:00
