Marown/yacy_search_server
1
0
Fork 0
mirror of https://github.com/yacy/yacy_search_server.git synced 2024-09-21 00:00:13 +02:00
Code Issues Actions Packages Projects Releases Wiki Activity
10,059 Commits 1 Branch 20 Tags 836 MiB
4684330505
Commit Graph

435 Commits

Author SHA1 Message Date
Michael Peter Christen
6905182d41 - fix for number of words log message 
- adding meta:refresh also to crawler stack
 2012-10-29 21:42:31 +01:00
Michael Peter Christen
a33e2742cb - removed unnecessary synchronized and deadlock in crawler 
- removed problem with monitoring object on Balancer.wait
- added missing user agent settings
 2012-10-28 19:56:02 +01:00
reger
722a447b0d - optimize code of augmented parsing to enhence document tags 
- commented out augmentedparser.analyse (not function implemented yet)
- adjust init of document title list to always use same list type
 2012-10-26 18:50:45 +02:00
orbiter
276dd6452b removed warnings 2012-10-23 19:08:44 +02:00
Michael Peter Christen
b991685782 Merge branch 'master' of git://gitorious.org/~reger/yacy/bbyacy-rc1 2012-10-23 18:14:58 +02:00
Michael Peter Christen
b7ac1da6a3 gsa results shall have only one title in metadata and that should be the 
visible title in the <title>-tag
 2012-10-23 18:03:12 +02:00
reger
87aab9aa7c - fix: with augmented parsing = on; missing metadata in index (like title) due to overwriting metadata by adding multiple result docs from augmentparser with same url 
- fix Document.addsubdocuments: sections might be initialized as Arrays.toList which does not provide the used .addAll methode
   see e.g. http://kamleshkr.wordpress.com/2010/02/17/inside-java-arrays-aslistt-a/
 2012-10-22 22:48:35 +02:00
Michael Peter Christen
ccc3760a47 Refactoring and redesign of data architecture to make URIMetadataRow 
superfluous. The target is to make a solr document as the core of YaCy
documents which would cause that many conversions can be removed. On the
way to this target the Equivalence of URIMetadataRow and URIMetadataNode
had to be removed to expose the usage of the old URIMetadataRow data
structure.
This refactoring already removes unneccessary conversions and should
make memory usage during indexing lower.
 2012-10-18 14:29:11 +02:00
Michael Peter Christen
21fe8339b4 - enhanced generation of url objects 
- enhanced computation of link structure graphics
- enhanced collection of data for link structures
 2012-10-15 13:17:13 +02:00
Michael Peter Christen
5f0ab25382 removed the option to prevent removal of &amp; parts inside of the 
MultiProtocolURI during normalform computation because that should
always be done and also be done during initialization of the
MultiProtocolURI Object. The new normalform method takes only one
argument which should be 'true' unless you know exactly what you are
doing.
 2012-10-10 11:46:22 +02:00
orbiter
68d0f8de03 Merge branch 'master' of git://gitorious.org/~reger/yacy/bbyacy-rc1 2012-10-09 20:36:32 +02:00
reger
bfb0d4c69b - add language detection from <html lang="xx"> tag 
- add jaudiotagger jar to Netbeans-IDE project classpath
 2012-10-09 20:02:58 +02:00
Michael Peter Christen
7e3e45fd04 added Open Graph Metadata default fields, see http://ogp.me/ns# 2012-10-09 17:28:48 +02:00
Michael Peter Christen
c3e5f667a7 added schema.org breadcrumb counter to parser and solr schema 2012-10-09 13:02:43 +02:00
Michael Peter Christen
4b5e0c1500 added an url rewriter which can be used to remove session ids from urls 2012-10-09 11:24:48 +02:00
Michael Peter Christen
584663ae8c - redesign of solr query construction 
- fix for solr boosts and location search
- fix for number of search results in local search
 2012-10-07 07:46:55 +02:00
Michael Peter Christen
6ab64746d7 Merge branch 'master' of ssh://git@gitorious.org/yacy/rc1.git 2012-10-06 03:35:32 +02:00
sof
5cb244b79b Merge remote branch 'origin/master' 2012-10-05 18:54:39 +02:00
apfelmaennchen
88b062210c Added a parser for audio file tags (e.g. ID3 tags for MP3 files) based 
on the jaudiotagger library. The parser is disabled by default as it
needs to store temporary files for non file:// protocols, which might be
disliked. For your local MP3-collection it loads nicely Artist,
Title, Album etc. from the audio files meta data.
 2012-10-05 18:54:26 +02:00
Michael Peter Christen
31485a963d refactoring 2012-10-02 21:57:50 +02:00
Michael Peter Christen
3d33a5bdf6 turned the synonyms_t Text field into a multi-valued String field 
synonyms_sxt
 2012-10-02 11:13:06 +02:00
Michael Peter Christen
3b959ee002 Merge branch 'master' of ssh://git@gitorious.org/yacy/rc1.git 2012-10-02 10:14:09 +02:00
orbiter
3190347814 added a synonyms_t field to solr and a process to read synonym files. 
This can be used to add another stemming to solr using stemming files
that are expressed as synonyms for grammatical alternatives. The
synonym/stemming files must have the following form:
- each line is a comma-separated list of synonyms
- the list of synonyms may be enclosed with {} (like the GSA synonyms
file)
- the file may contain comments which are lines starting with a '#'
The synonym file(s) must be placed in DATA/DICTIONARIES/synonyms/ and
are activated by default whenever a synonym file is in place.
Then, for each word that is found in a document all synonyms are added
to a long text field which is stored into synonyms_t. Processes using
the synonyms must query with that field as optional matcher.
 2012-10-02 00:02:50 +02:00
Michael Peter Christen
411d0e839b added an underline text field to solr to record all underlined texts 2012-10-01 14:16:49 +02:00
Michael Peter Christen
24d2ee3c52 - better date ranking 
- more protection against NPE and time travel effects
 2012-09-26 18:36:32 +02:00
sixcooler
6c50d016ed pdf- and zipParser should not use forced Memory-Limits 2012-09-26 14:03:51 +02:00
Michael Peter Christen
1533bfd63b refactoring 2012-09-25 21:20:03 +02:00
Michael Peter Christen
8219a445f3 refactoring 2012-09-21 16:46:57 +02:00
Michael Peter Christen
00c1c777fa refactoring 2012-09-21 15:48:16 +02:00
orbiter
63762d8f89 removed kelondro dependencies from cora 2012-09-20 19:38:22 +02:00
Michael Peter Christen
e54ac38095 - some corrections in usage of getFile() and getFileName() 
- added more attributes in json response writer according to yacy
servlet
 2012-09-11 23:28:21 +02:00
Michael Peter Christen
528d6763fa - added new solr fields: 
title_count_i, title_chars_val, title_words_val
description_count_i, description_chars_val, description_words_val
- added many asserts to ensure data type correctness from YaCy to Solr
and vice versa
- made many fixes according to new findings from these asserts (!)
 2012-08-31 10:30:43 +02:00
Michael Peter Christen
e8acd542b5 - added faceted drill-down for host and geolocation to solr queries 
- added a new geolocation field to index schema, the old values are
migrated if possible
 2012-08-27 14:41:33 +02:00
orbiter
67f2866cd0 small fixes 2012-08-24 21:44:22 +02:00
orbiter
67edfd991c Merge branch 'master' of ssh://git@gitorious.org/yacy/rc1.git 2012-08-05 15:49:48 +02:00
orbiter
d9173ba7ed added more solr fields to integrate values from URIMetadataRow. All 
writings to the Metadata-DB are now also done to solr. This includes
metadata transfer during search and rwi transfer.

The new/added solr fields are:

## time when resource was loaded
load_date_dt

## date until resource shall be considered as fresh
fresh_date_dt

## id of the host, a 6-byte hash that is part of the document id
host_id_s

## ids of referrer to this document
referrer_id_ss

## the md5 of the raw source
md5_s

## the name of the publisher of the document
publisher_t

## the language used in the document; starts with primary language
language_ss

## an external ranking value
ranking_i

## the size of the raw source
size_i

## number of links to audio resources
audiolinkscount_i

## number of links to video resources
videolinkscount_i

## number of links to application resources
applinkscount_i
 2012-08-05 15:49:27 +02:00
Michael Peter Christen
24d9db1613 snippet retrieval loading processes may use a smaller minimum load time 
value than crawling processes. This speeds up the search result
preparation dramatically.
 2012-07-30 10:38:23 +02:00
Michael Peter Christen
1687737771 Abstraction of HandleMap and HandleSet 2012-07-27 12:13:53 +02:00
orbiter
482afed07c reduced logging overhead (a bit) 2012-07-12 19:23:40 +02:00
orbiter
bbfa497a3c replaced more size() > 0 by !isEmpty() 2012-07-12 11:12:21 +02:00
orbiter
0cbda0b2b8 - replaced all length() == 0 and size() == 0 with isEmpty() 
- replaced some length() > 0 and size() > 0 with !isEmpty() - cannot be
done automatically
- implemented some isEmpty() methods
 2012-07-10 22:59:03 +02:00
Michael Peter Christen
801972fe6f fix for url camel case parser and sentence reader 2012-07-08 16:48:09 +02:00
Michael Peter Christen
fbc1a2030d fix for sitemap importer: can now also import very large sitemaps within 
small memory configurations
 2012-07-08 16:11:50 +02:00
Michael Peter Christen
92731e5287 fix for sevenzip parser 2012-07-08 16:11:19 +02:00
Michael Peter Christen
8efc1c1078 - fixed a memory leak (or bad usage) during parsing/snippet fetch 
- more logging for errors
 2012-07-06 09:05:41 +02:00
Michael Peter Christen
b1e7c11fba fix for pattern matcher in html parser 2012-07-05 14:24:03 +02:00
Michael Peter Christen
b0c408788b made class methods static where possible 2012-07-05 12:38:41 +02:00
Michael Peter Christen
7c1ba99755 removed more unused method parameters 2012-07-05 10:44:30 +02:00
Michael Peter Christen
0301aba1e9 removed unused method parameters 2012-07-05 10:23:07 +02:00
Michael Peter Christen
d3964253ae - added @SuppressWarnings to unused servlet method parameters 
- removed unnecessary casts
- removed unnecessary throw statements
 2012-07-05 09:14:04 +02:00
Michael Peter Christen
ea10766bfd cleaned unnecessary nested code 2012-07-05 08:44:39 +02:00
orbiter
fc0f9543fe More SentenceReader cleanup 2012-07-05 00:20:58 +02:00
orbiter
586bb0eb6a Simplified SentenceReader (no more Reader inside..) 2012-07-04 22:06:20 +02:00
orbiter
7f851d62a7 replaced HashARC with SizeLimited Objects which are less costly 2012-07-04 21:56:25 +02:00
orbiter
78fc3cf8f8 refactoring and new usage of SentenceReader: this class appeared as one 
of the major CPU users during snippet verification. The class was not
efficient for two reasons:
- it used a too complex input stream; generated from sources and UTF8
byte-conversions. The BufferedReader applied a strong overhead.
- to feed data into the SentenceReader, multiple toString/getBytes had
been applied until a buffered Reader from an input stream was possible.
These superfluous conversions had been removed.
- the best source for the Sentence Reader is a String. Therefore the
production of Strings had been forced inside the Document class.
 2012-07-04 21:15:10 +02:00
orbiter
bb8dcb4911 automatically adopt size of word cache to available memory 2012-07-03 18:22:25 +02:00
Michael Peter Christen
ad09b786bf clean up parser data 2012-07-03 17:20:41 +02:00
Michael Peter Christen
276a66a793 Adding a limit of 1000 links that a parser shall store during indexing. 
A limit was necessary because some web pages have such huge numbers of
links that it can easily cause a OOM just by the number of links.
The quesion if the number of 1000 links is sufficient or too weak must
be answered with the result of testing this feature.
 2012-07-03 17:06:20 +02:00
Michael Peter Christen
de903a53a0 parser refactoring & hacks 2012-07-03 06:06:38 +02:00
Michael Peter Christen
1825f165b8 better integration of blacklist according to use case 2012-07-02 13:57:29 +02:00
Michael Peter Christen
ce8d4b87d9 fixes for new eclipse 'Juno' warning 'Resource leak'. 2012-07-02 10:27:46 +02:00
Michael Peter Christen
0c345d1559 giving threads name so its easier to see whats happening during 
debugging and within a thread dump
 2012-07-02 09:51:43 +02:00
Michael Peter Christen
508a81b86c added solr field 'refresh_s' which stores the refresh url contained in 
the meta-refresh html header field.
 2012-06-28 13:27:45 +02:00
Michael Peter Christen
f3167def64 do not fill the keywords with title content if keywords do not exist. 2012-06-27 13:07:02 +02:00
Michael Peter Christen
77f795756c fixing redirects and status codes: storing of status code in 
ResponseHeader to make it available for late evaluations, like storage
in solr.
 2012-06-25 18:17:31 +02:00
Michael Peter Christen
dbdd697f4d moved RDFaParser.xsl configuration file to defaults 2012-06-21 16:09:12 +02:00
Michael Peter Christen
786be7d175 better integration of RDFaParser 2012-06-20 16:39:04 +02:00
Michael Peter Christen
de3ef8ad73 removed unimportant warnings 2012-06-19 08:45:34 +02:00
Michael Peter Christen
24bbe359ca integrate also geonames library files for less cities. these are more 
useful for tagging since less normal words are false-identified as
location
 2012-06-18 15:19:57 +02:00
Michael Peter Christen
223a5440ab preventing that an empty pnd is inserted into the vocabularies 2012-06-18 01:22:39 +02:00
Michael Peter Christen
963f92ed9a - merged files 
- changed behaviour of delete button in vocabulary edit
- fixed size numbe in vocabulary listing
 2012-06-17 23:48:33 +02:00
Michael Peter Christen
dd88d0ace2 more logging 2012-06-17 19:03:53 +02:00
Michael Peter Christen
94d54e2d91 added recognition of multi-word terms in vocabulary matching 
this makes the PND usable: it is now possible to recognize persons and
navigate with a 'Persons' facet.
 2012-06-16 19:40:27 +02:00
Michael Peter Christen
64c0268b2b show triplestore metadata in yacydoc and viewfile 2012-06-16 17:40:15 +02:00
Michael Peter Christen
c2f0d16d2c fixed vocabulary initialization 2012-06-16 13:12:02 +02:00
Michael Peter Christen
df3531f8d5 added the generation of virtual vocabularies using the pnd 2012-06-16 12:36:15 +02:00
Michael Peter Christen
a0f1decd82 - added loading of the dbpedia pnd triplestore in the dictionary loader 
- renamed the dictionary loader to knowledge loader
- some refactoring in the library provider method names
 2012-06-15 19:19:18 +02:00
Michael Peter Christen
16d8f33795 added objectlink generation to vocabulary generation and editor 2012-06-14 18:50:35 +02:00
Michael Peter Christen
d45718251e refactoring (Localization -> Location) 2012-06-14 09:45:57 +02:00
Michael Peter Christen
b8b3c87ba7 - renamed localization to location (that was confusing) 
- renamed 'Locale' navigator to 'Location'
- produce Location navigation only if geolocation libraries are loaded
 2012-06-14 09:44:14 +02:00
Michael Peter Christen
e89747bb67 - added automated generation of vocabularies from url stubs 
- added clear of all terms for vocabularies
- added deletion of vocabularies
 2012-06-13 15:53:18 +02:00
Michael Peter Christen
79464189a4 The 'Locale' vocabulary, which is generated by geo data, has now the 
objectspace "http://dbpedia.org/resource/"
 2012-06-13 13:05:41 +02:00
Michael Peter Christen
61bb52d55c - using http://purl.org/dc/terms/references to refer from an 
auto-annotated document to a 'pseudo-linked' document which has an url
created with an object-prefix as defined in the vocabulary file
 2012-06-12 14:23:51 +02:00
Michael Peter Christen
50c576599b allow multiple parser options instead of printing an error 2012-06-12 01:42:58 +02:00
Michael Peter Christen
8b53771db2 changed behavior of navigation processing: 
- vocabulary annotation is not done any more into the metadata of urldb
- vocabularies are written into the jena triplestore using a rdf
vocabulary
- vocabularies for rdf tripel must be updated; refactoring done
- with the new navigation tags in the triplestore a faster
pre-urldb-lookup is possible: navigation is processed now within the RWI
during pre-ranking retrieval
- added also a Owl vocabulary stub to add the plain-text url to the
triplestore using the owl:sameas predicate
 2012-06-11 23:49:30 +02:00
Michael Peter Christen
5fc6524ca8 - moved triple store to net.yacy.cora.lod (should be generalized there 
later
- added abstract add, delete, get methods in the triplestore
- added generation of triples after auto-annotation
- migrated all MultiProtocolURI objects to DigestURI in the parser since
the url hash is needed as subject value in the triples in the triple
store
 2012-06-11 16:48:53 +02:00
cominch
bbfc53b663 bugfix 2012-06-10 13:12:12 +02:00
cominch
65c5826d93 bugfix 
Conflicts:
	source/net/yacy/document/parser/augment/AugmentParser.java
 2012-06-10 13:11:54 +02:00
cominch
5f8ba7f4f2 small changes 
Conflicts:
	source/net/yacy/document/parser/augment/AugmentParser.java
	source/net/yacy/interaction/Interaction.java
 2012-06-10 13:02:00 +02:00
cominch
90512640bf Added config switches for custom parser 
Conflicts:
	source/net/yacy/document/TextParser.java
 2012-06-10 12:49:36 +02:00
cominch
bcbd8eee33 Add several parsers, for RDFa and rdf files. 
Conflicts:
	source/net/yacy/document/TextParser.java
 2012-06-10 10:42:33 +02:00
cominch
9cbfc1a1c0 augmentedProxy, which forwards every proxy request to a 
rewrite engine to customize existing webpages. originally implemented by
Florian Richter.

Conflicts:
	source/de/anomic/http/server/HTTPDProxyHandler.java
 2012-06-10 10:15:34 +02:00
Michael Peter Christen
cde20911bb saved a bit more ram using UTF8 String compression for OpenGeoDB and 
Geonames data files.
 2012-06-09 10:07:11 +02:00
Michael Peter Christen
225ee42879 made the GeoLocation into an interface with the current 
integer implementation as accuracy implementation of 1.863cm
 2012-06-09 09:46:27 +02:00
Michael Peter Christen
96e9d77270 Merge branch 'master' of ssh://git@gitorious.org/yacy/rc1.git 
Conflicts:
	source/net/yacy/cora/sorting/WeakPriorityBlockingQueue.java
 2012-06-06 20:13:28 +02:00
Michael Peter Christen
96c8119b50 added GeoLocation / GeoPoint classes which uses less memory than 
Location/Coordinates and has initializers with correct order of lat,lon
coordinates
 2012-06-06 12:57:42 +02:00
Michael Peter Christen
461a0ce052 removed warnings 2012-06-05 20:03:43 +02:00
Michael Peter Christen
2fe207f813 Merge branch 'master' of ssh://git@gitorious.org/yacy/rc1.git 2012-06-04 23:44:38 +02:00
Michael Peter Christen
514700291a moved Vocabulary to cora package (added in git 
964406ad17)
 2012-06-04 23:41:36 +02:00
Michael Peter Christen
0284a4d88f more fixes for double precision of coordinates 2012-06-04 23:37:41 +02:00