Marown/yacy_search_server
1
0
Fork 0
mirror of https://github.com/yacy/yacy_search_server.git synced 2024-09-21 00:00:13 +02:00
Code Issues Actions Packages Projects Releases Wiki Activity
8,766 Commits 1 Branch 20 Tags 836 MiB
b3aad6cc35
Commit Graph

5766 Commits

Author SHA1 Message Date
Michael Peter Christen
b3aad6cc35 bugfix for remote search when search is done to solr 2012-08-20 12:21:36 +02:00
Michael Peter Christen
ff3eaa21b0 added remote search to solr on YaCy peers! 
- when doing a remote search, node peers are selected for solr queries
- the solr query is done concurrently to the standard YaCy rwi search
- the solr search result is feeded into the same data structure that
prepares the rwi search result
- the same remote seach that is done to several outside peers is done to
the local solr index
- the search process works now also without any 'old' RWI data using
solr
 2012-08-20 12:16:11 +02:00
Michael Peter Christen
a06123aec6 more abstraction and less parameter overhead for remote search 2012-08-20 01:29:15 +02:00
Michael Peter Christen
f00733186b code simplifications 2012-08-19 13:17:03 +02:00
Michael Peter Christen
755f5e76cf removed strange assert statements and simplified code in metadata 
transformation
 2012-08-19 08:44:39 +02:00
Michael Peter Christen
db0d438709 fix for http://bugs.yacy.net/view.php?id=206 2012-08-19 08:43:56 +02:00
orbiter
404b0aab09 refactoring in remote search and stub for remote node peer selection 2012-08-18 23:59:25 +02:00
orbiter
d7ea45f698 - get nice text_t values from metadata conversions that are stored into 
solr as fulltext search index.
- added slow migration from old metadata to solr index entries: each
entry from the old metadata is removed from that data structure and
written into solr.
 2012-08-18 19:36:21 +02:00
orbiter
99ef57f103 reduced sleep times 2012-08-18 17:48:20 +02:00
orbiter
780f8974e7 added ramaining iteration methods for solr in fulltext class 2012-08-18 15:39:14 +02:00
orbiter
acd2dc3575 hack to removed StringBuilder overhead in query construction 2012-08-18 14:22:00 +02:00
orbiter
ee01c12e56 fixes for putDocument and putMetadata 2012-08-18 13:05:27 +02:00
orbiter
cc47a0876e reverted bf55f69176 
to have a fall-back option in case that memory problems as reported in
http://forum.yacy-websuche.de/viewtopic.php?p=26901#p26901
for full-solr installation are too strong and we have to work with an
'small memory footprint' peer system.
 2012-08-18 10:28:40 +02:00
Michael Peter Christen
0904afe8fb added concurrent iterator methods to the solr connectors 2012-08-17 18:22:56 +02:00
Michael Peter Christen
d54b80327a refactoring 2012-08-17 17:28:27 +02:00
Michael Peter Christen
f9fc5cfaba better check for bad urls in url transmission 2012-08-17 17:17:00 +02:00
Michael Peter Christen
d39463a85c added deleteByQuery to solr connectors 2012-08-17 17:05:46 +02:00
Michael Peter Christen
0cab06c47c refactoring 2012-08-17 15:52:33 +02:00
Michael Peter Christen
bf55f69176 removed write methods to old metadata file type; all metadata now goes 
to solr
 2012-08-17 15:46:26 +02:00
Michael Peter Christen
40c0856489 refactoring 2012-08-17 15:33:02 +02:00
Michael Peter Christen
06a78eecb7 code simplification 2012-08-17 14:43:32 +02:00
Michael Peter Christen
54bea21c02 bugfix for solr connector, possibly a cause for 
http://forum.yacy-websuche.de/viewtopic.php?p=26893#p26893
 2012-08-17 14:34:31 +02:00
Michael Peter Christen
9bece5ac5f enhanced snippet fetch - removed a bug that caused documents to be 
parsed even if a solr text was available
 2012-08-17 14:22:07 +02:00
Michael Peter Christen
18f989dfb1 - refactoring (load -> getMetadata) 
- added getDocument to retrieve Solr documents which shall replace
getMetadata
 2012-08-17 01:34:38 +02:00
Michael Peter Christen
395b78a0d8 using the solr search index to concurrently search within solr and the 
rwis during local search requests.
 2012-08-17 01:21:56 +02:00
Michael Peter Christen
6197caf698 added clear-text search words in query params 2012-08-16 23:05:37 +02:00
Michael Peter Christen
efafa79db5 - added a content-encoding: gzip to streamed http server responses 
- finish and close streamed http responses immediately
- this applies only to the solr interface which should be much faster
now!
 2012-08-16 22:35:19 +02:00
Michael Peter Christen
23226676c6 FOR THE BRAVE.. this is a forced migration to solr which is now ready 
for production as a replacement of the metadata-db.
This intermediate release 1.041 will switch on the previously optional
solr index and the old metadata-db will still work as it did before.
Solr+metadata are accessed in mixed mode, no migration is done yet.
If this causes not a catastrophe until the end of the weekend, we will
do a YaCy 1.1 main release containing this as default.
 2012-08-16 18:17:47 +02:00
Michael Peter Christen
a1b2c9a67d doctype2mime fix, influences metadata conversion between old metadata 
and solr
 2012-08-16 17:49:35 +02:00
Michael Peter Christen
a16206e38b more attempts to clean the index (cleaning is faster then) 2012-08-16 17:24:25 +02:00
Michael Peter Christen
703f427303 fixed some peer-ping connection details 
- larger time-out
- removed too old seedlist
- fixed a bug in connection test
 2012-08-16 17:11:54 +02:00
Michael Peter Christen
597bb76e4f get the peer location more quickly 2012-08-16 16:28:57 +02:00
Michael Peter Christen
1641835fef replaced yacy xml encoding by solr xml encoding 2012-08-14 13:29:11 +02:00
Michael Peter Christen
89fe13e73d enhanced GSA and RSS output format: corrected date, added some missing 
fields, added xml encoding for utf8
 2012-08-14 13:19:29 +02:00
Michael Peter Christen
ea49a8aa8c Merge branch 'master' of ssh://git@gitorious.org/yacy/rc1.git 2012-08-14 12:40:44 +02:00
Michael Peter Christen
d988ba50cf added a very rudimentary, incomplete, non-verified GSA response writer 
for solr. Try this:
http://localhost:8090/gsa/searchresult?q=pdf&site=col1&num=10
 2012-08-14 12:40:26 +02:00
Michael Peter Christen
aab0b680c3 - added xslt support for solr result formats. 
try i.e.
http://localhost:8090/solr/select?q=*:*&start=0&rows=10&wt=xslt&tr=json.xsl
- added servlet-side mime-type configuration for streamed servlets. this
is used for the result formatters in solr result formats
 2012-08-14 11:12:50 +02:00
cominch
e2119f4e76 augmented browsing: replace htmlparser by jsoup, which is more stable 
and reliable
 2012-08-14 10:06:12 +02:00
Michael Peter Christen
9448d9a8a2 ups 2012-08-13 14:01:45 +02:00
Michael Peter Christen
e5ef840f40 - renamed DoubleSolrConnector to MirrorSolrConnector and added a 
hit/miss/document cache to the MirrorSolrConnector.
- more abstraction to SolrDocument in Connector interface
- bugfixes in Solr field reader
 2012-08-13 13:32:32 +02:00
Michael Peter Christen
94a334f128 another fix to the Solr metadata reading process and to the shutdown 
process
 2012-08-13 11:13:53 +02:00
Michael Peter Christen
b51df6c7e8 - added coordinate storage in solr schema 
- fixed shutdown process
- fixed some solr-to-metadata reading
- added a large number of metadata attributes in ViewFile.html
 2012-08-13 10:40:04 +02:00
Michael Peter Christen
da851c6071 Merge branch 'master' of ssh://git@gitorious.org/yacy/rc1.git 2012-08-11 01:21:18 +02:00
Michael Peter Christen
bd4f03bc85 removed unused class 2012-08-11 01:05:40 +02:00
orbiter
39f8eb60c3 tried to prevent calls to bad-hack getSize() method and reduced overhead 
of that method a bit.
 2012-08-10 18:10:25 +02:00
orbiter
e816b88b55 changed behaviour of metadata storage: in case that any solr is 
attached, the metadata is not written to the metadata-db, even if it is
enabled but instead to solr. This prevents that metadata is written in
two store systems at the same time. It is also the next step to migrate
the current metadata-db to solr.
 2012-08-10 15:39:10 +02:00
orbiter
2571e0d47a removed unused classes 2012-08-10 14:47:44 +02:00
Michael Peter Christen
f9c0e6e950 - Implemented and integrated the URIMetadataNode object which is a 
metadata representation from the solr index. This shall replace metadata
from the built-in database in the future.
- added the Solr-driven metadata into the search index of YaCy which
makes it now possible to run YaCy without the old metadata index. This
is a major stept forward to a full migration to Solr.
 2012-08-10 13:26:51 +02:00
Michael Peter Christen
b2b480fff2 more abstraction of the YaCySchema -> Opensearch matching process 2012-08-10 09:48:15 +02:00
Michael Peter Christen
24462e9baa set the title every time, it is possible that it has changed 2012-08-10 07:51:57 +02:00