Marown/yacy_search_server
1
0
Fork 0
mirror of https://github.com/yacy/yacy_search_server.git synced 2024-09-21 00:00:13 +02:00
Code Issues Actions Packages Projects Releases Wiki Activity
8,604 Commits 1 Branch 20 Tags 836 MiB
241dd8410a
Commit Graph

1227 Commits

Author SHA1 Message Date
Michael Peter Christen
7dc59979bc fix for npe, possibly for http://bugs.yacy.net/view.php?id=195 2012-06-18 21:25:39 +02:00
Michael Peter Christen
1d4e206b2b bugfix in vocabulary generation 2012-06-18 18:10:40 +02:00
cominch
2c89975378 Merge remote-tracking branch 'original yacy/master' 2012-06-18 16:16:46 +02:00
Michael Peter Christen
52f5d40043 better abstraction of document model generation 2012-06-18 15:55:18 +02:00
Michael Peter Christen
8b7c4d3144 produce a rdf output containing the triplestore with yacydoc; ie: 
http://localhost:8090/api/yacydoc.rdf?urlhash=yOiCM7Fh1hyQ
 2012-06-18 15:47:54 +02:00
cominch
f7160dae5c Merge remote-tracking branch 'original yacy/master' 2012-06-18 15:44:50 +02:00
cominch
e4555cbee3 Augmented browsing: Pass on additional action parameter 2012-06-18 15:44:01 +02:00
Michael Peter Christen
24bbe359ca integrate also geonames library files for less cities. these are more 
useful for tagging since less normal words are false-identified as
location
 2012-06-18 15:19:57 +02:00
Michael Peter Christen
223a5440ab preventing that an empty pnd is inserted into the vocabularies 2012-06-18 01:22:39 +02:00
Michael Peter Christen
8e97ada7c9 IPv6 bugfix 2012-06-18 00:33:32 +02:00
Michael Peter Christen
963f92ed9a - merged files 
- changed behaviour of delete button in vocabulary edit
- fixed size numbe in vocabulary listing
 2012-06-17 23:48:33 +02:00
Michael Peter Christen
dd88d0ace2 more logging 2012-06-17 19:03:53 +02:00
Michael Peter Christen
94d54e2d91 added recognition of multi-word terms in vocabulary matching 
this makes the PND usable: it is now possible to recognize persons and
navigate with a 'Persons' facet.
 2012-06-16 19:40:27 +02:00
Michael Peter Christen
64c0268b2b show triplestore metadata in yacydoc and viewfile 2012-06-16 17:40:15 +02:00
Michael Peter Christen
0fbd749207 ipv6 update 2012-06-16 15:57:00 +02:00
Michael Peter Christen
c2f0d16d2c fixed vocabulary initialization 2012-06-16 13:12:02 +02:00
Michael Peter Christen
df3531f8d5 added the generation of virtual vocabularies using the pnd 2012-06-16 12:36:15 +02:00
Michael Peter Christen
a0f1decd82 - added loading of the dbpedia pnd triplestore in the dictionary loader 
- renamed the dictionary loader to knowledge loader
- some refactoring in the library provider method names
 2012-06-15 19:19:18 +02:00
cominch
2ac7a5c1f2 Augmented browsing: Add overlay bar which shows the vocabulary tags 2012-06-15 14:32:16 +02:00
cominch
3c255c025b Show tags in search results (if activated in ConfigPortal_p.html) 2012-06-15 10:43:05 +02:00
Michael Peter Christen
16d8f33795 added objectlink generation to vocabulary generation and editor 2012-06-14 18:50:35 +02:00
cominch
f49d92d8da Cleanup of interaction class and helper routines 2012-06-14 17:41:45 +02:00
cominch
56b0115054 Triplestore: modify routines to access per user store 2012-06-14 15:44:27 +02:00
cominch
a95127c9af Triplestore: initalize per-user triplestores 2012-06-14 11:46:53 +02:00
Michael Peter Christen
d45718251e refactoring (Localization -> Location) 2012-06-14 09:45:57 +02:00
Michael Peter Christen
b8b3c87ba7 - renamed localization to location (that was confusing) 
- renamed 'Locale' navigator to 'Location'
- produce Location navigation only if geolocation libraries are loaded
 2012-06-14 09:44:14 +02:00
Michael Peter Christen
e89747bb67 - added automated generation of vocabularies from url stubs 
- added clear of all terms for vocabularies
- added deletion of vocabularies
 2012-06-13 15:53:18 +02:00
Michael Peter Christen
79464189a4 The 'Locale' vocabulary, which is generated by geo data, has now the 
objectspace "http://dbpedia.org/resource/"
 2012-06-13 13:05:41 +02:00
Michael Peter Christen
eca38c53e7 added a vocabulary editor 2012-06-13 12:12:20 +02:00
Michael Peter Christen
61bb52d55c - using http://purl.org/dc/terms/references to refer from an 
auto-annotated document to a 'pseudo-linked' document which has an url
created with an object-prefix as defined in the vocabulary file
 2012-06-12 14:23:51 +02:00
Michael Peter Christen
2bbb6c52cf added option to clean the triplestore when deleting the index 2012-06-12 01:54:36 +02:00
Michael Peter Christen
50c576599b allow multiple parser options instead of printing an error 2012-06-12 01:42:58 +02:00
Michael Peter Christen
c02d742e53 proper namespaces in triplestore dump 2012-06-12 00:20:11 +02:00
Michael Peter Christen
8b53771db2 changed behavior of navigation processing: 
- vocabulary annotation is not done any more into the metadata of urldb
- vocabularies are written into the jena triplestore using a rdf
vocabulary
- vocabularies for rdf tripel must be updated; refactoring done
- with the new navigation tags in the triplestore a faster
pre-urldb-lookup is possible: navigation is processed now within the RWI
during pre-ranking retrieval
- added also a Owl vocabulary stub to add the plain-text url to the
triplestore using the owl:sameas predicate
 2012-06-11 23:49:30 +02:00
Michael Peter Christen
5fc6524ca8 - moved triple store to net.yacy.cora.lod (should be generalized there 
later
- added abstract add, delete, get methods in the triplestore
- added generation of triples after auto-annotation
- migrated all MultiProtocolURI objects to DigestURI in the parser since
the url hash is needed as subject value in the triples in the triple
store
 2012-06-11 16:48:53 +02:00
cominch
8d2e6355f8 augmented browsing: remove non-existing external snippet file 2012-06-11 11:40:48 +02:00
cominch
c90f174799 preparation and generalization of augmented browsing methods 2012-06-11 09:23:44 +02:00
Michael Peter Christen
bef823c247 close the reader if finished 2012-06-11 01:20:54 +02:00
Michael Peter Christen
4ee6fb1de9 added missing blacklist dht cache storage (maybe due to mistakes in 
cherry picking)
 2012-06-11 00:38:02 +02:00
Roland 'Quix0r' Haeder
e4d36fa5eb Fix to make all values lower-case (this should make all existing blacklists compatible with the new enum) 2012-06-11 00:17:53 +02:00
Roland 'Quix0r' Haeder
edaa09b9b1 Rewrote all String blacklist types to enum 'BlacklistType', closes bug 
#143

Conflicts:
	htroot/Supporter.java
	htroot/yacy/crawlReceipt.java
	htroot/yacy/transferRWI.java
	htroot/yacy/transferURL.java
	source/de/anomic/crawler/CrawlStacker.java
	source/de/anomic/data/ListManager.java
	source/net/yacy/peers/Protocol.java
	source/net/yacy/repository/Blacklist.java
	source/net/yacy/repository/LoaderDispatcher.java
	source/net/yacy/search/Switchboard.java
	source/net/yacy/search/index/MetadataRepository.java
	source/net/yacy/search/index/Segment.java
	source/net/yacy/search/query/RWIProcess.java
	source/net/yacy/search/snippet/MediaSnippet.java
 2012-06-11 00:17:30 +02:00
Roland 'Quix0r' Haeder
af5a597e47 Scroogle is not comming back, remove dead code 
Conflicts:
	source/net/yacy/search/Switchboard.java
 2012-06-10 23:38:41 +02:00
Michael Peter Christen
ca93835713 removed usage of deprecated methods 2012-06-10 23:17:21 +02:00
Michael Peter Christen
23e38bd918 do not load the "_triplestore.rdf" files which are of special use in 
sciencenet
 2012-06-10 21:54:30 +02:00
Michael Peter Christen
90c6fc4b63 load all - but not the persistent local.rdf - triples from 
DATA/TRIPLESTORE at startup time. The local.rdf is loaded only if the
persistent switch is on (as before).
 2012-06-10 21:49:02 +02:00
cominch
bbfc53b663 bugfix 2012-06-10 13:12:12 +02:00
cominch
65c5826d93 bugfix 
Conflicts:
	source/net/yacy/document/parser/augment/AugmentParser.java
 2012-06-10 13:11:54 +02:00
cominch
aa0295917c augmentation 
Conflicts:
	source/net/yacy/interaction/AugmentHtmlStream.java
 2012-06-10 13:10:21 +02:00
cominch
ed2ea0f08e augmented browsing modification 
Conflicts:
	htroot/interaction/OverlayInteraction.html
	source/net/yacy/interaction/AugmentHtmlStream.java
 2012-06-10 13:07:57 +02:00
cominch
6b32f7c1f6 re-enable augmented proxy 2012-06-10 13:04:13 +02:00
cominch
3b08edec2e bugfix 
Conflicts:
	source/net/yacy/interaction/AugmentHtmlStream.java
 2012-06-10 13:03:56 +02:00
cominch
5f8ba7f4f2 small changes 
Conflicts:
	source/net/yacy/document/parser/augment/AugmentParser.java
	source/net/yacy/interaction/Interaction.java
 2012-06-10 13:02:00 +02:00
cominch
300b235ce8 Updated Demo Servlet 
Conflicts:
	htroot/About.html
	htroot/DemoServlet.html
	htroot/DemoServlet.java
	htroot/interaction/interaction.js
	source/net/yacy/interaction/Interaction.java
 2012-06-10 12:58:29 +02:00
cominch
90512640bf Added config switches for custom parser 
Conflicts:
	source/net/yacy/document/TextParser.java
 2012-06-10 12:49:36 +02:00
cominch
df47f31235 interaction: add special table interaction 
Conflicts:
	source/net/yacy/interaction/Interaction.java
 2012-06-10 12:43:01 +02:00
cominch
e14f2881ae interaction: add special table interaction 
Conflicts:
	source/net/yacy/interaction/Interaction.java
 2012-06-10 12:41:16 +02:00
cominch
d7326079a8 interaction: add global variable store 
Conflicts:
	source/net/yacy/interaction/Interaction.java
 2012-06-10 12:39:27 +02:00
cominch
4e4e7a99f8 interaction: add global variable store 
Conflicts:
	source/net/yacy/interaction/Interaction.java
 2012-06-10 12:34:36 +02:00
cominch
bde07ed7a8 Add tagging overlay element 
Conflicts:
	htroot/env/templates/jqueryheader.template
	htroot/yacysearchitem.java
	source/net/yacy/interaction/Interaction.java
 2012-06-10 12:28:50 +02:00
cominch
b0bc0b4572 Add new demonstration module for client-side key-value store (backend: 
triplestore): /DemoServletInteraction.html

Conflicts:
	source/net/yacy/interaction/Interaction.java
 2012-06-10 10:53:30 +02:00
cominch
c9dc6cda02 Demonstration: include value from interaction in search results 
Conflicts:
	htroot/interaction/OverlayInteraction.html
	htroot/yacysearchitem.java
 2012-06-10 10:51:53 +02:00
cominch
bcbd8eee33 Add several parsers, for RDFa and rdf files. 
Conflicts:
	source/net/yacy/document/TextParser.java
 2012-06-10 10:42:33 +02:00
cominch
9ef5a80f4e add interaction for triples and selector for augmented browsing 
Conflicts:
	htroot/interaction/interaction.js
	source/net/yacy/interaction/Interaction.java
 2012-06-10 10:38:54 +02:00
cominch
282c1620d6 Allow TripleStore to be persistent after reboot 2012-06-10 10:36:16 +02:00
cominch
5d20cd324a Add Triplestore and RDF query interface 
Conflicts:
	build.xml
	defaults/yacy.init
	source/net/yacy/interaction/AugmentHtmlStream.java
 2012-06-10 10:35:59 +02:00
cominch
bc9a618e0a augmented browsing: ignore js and css, integrate more user interaction 
Conflicts:
	htroot/interaction/Footer.html
	source/net/yacy/interaction/AugmentHtmlStream.java
 2012-06-10 10:29:15 +02:00
cominch
b21048892b augmentedParser add features and integrate external html parser to 
modify existing web pages

Conflicts:
	addon/YaCy.app/Contents/Info.plist
	build.xml
 2012-06-10 10:23:35 +02:00
cominch
9cbfc1a1c0 augmentedProxy, which forwards every proxy request to a 
rewrite engine to customize existing webpages. originally implemented by
Florian Richter.

Conflicts:
	source/de/anomic/http/server/HTTPDProxyHandler.java
 2012-06-10 10:15:34 +02:00
Michael Peter Christen
3b992e6b00 using utf8 String compression in Webstructure database 2012-06-09 11:00:33 +02:00
Michael Peter Christen
26301a538d bugfix in Domains - dns-lookup 2012-06-09 10:59:45 +02:00
Michael Peter Christen
cde20911bb saved a bit more ram using UTF8 String compression for OpenGeoDB and 
Geonames data files.
 2012-06-09 10:07:11 +02:00
Michael Peter Christen
225ee42879 made the GeoLocation into an interface with the current 
integer implementation as accuracy implementation of 1.863cm
 2012-06-09 09:46:27 +02:00
Michael Peter Christen
2280a7b276 - changed initialization order to prefer allocation of memory for table 
files first
- bugfixes in memory amount calculation
 2012-06-09 09:05:47 +02:00
Michael Peter Christen
0746308bc2 only the metadata tables shall be able to use the tail cache 2012-06-08 18:36:11 +02:00
Michael Peter Christen
7ec9bef0c3 fix for OOM 2012-06-08 17:14:09 +02:00
Michael Peter Christen
41c02cb10e - less restrictions for usage of Table RAM copy 
- new limit to use the table copy (instead of flag): 400MB available. If
less is available, then a copy is never used. If more is available, then
it can be used if there is a remaining space of at least 200MB
- flush caches more often: flush the Digest cache
 2012-06-08 12:48:25 +02:00
Michael Peter Christen
b8f56a9803 npe bugfix 2012-06-08 10:20:43 +02:00
Michael Peter Christen
dd14b19c26 lazy initialization of block rank table ... only normal web search uses 
this. When interactive search or location search is used, the block rank
is switched off
 2012-06-08 09:41:29 +02:00
Michael Peter Christen
ba10caf89a lazy initialization of database tables 2012-06-08 09:30:51 +02:00
Michael Peter Christen
701b9a28a0 Merge branch 'master' of ssh://git@gitorious.org/yacy/rc1.git 
Conflicts:
	htroot/PerformanceMemory_p.java
 2012-06-08 09:16:16 +02:00
Michael Peter Christen
ab7107b34b fixed RWIProcess queue limits: now discovering hidden results for mass 
result retrieval
 2012-06-08 09:14:54 +02:00
Michael Peter Christen
10c9c17d51 fixed handlemap spread factor and null iterator handling 2012-06-08 09:13:41 +02:00
Michael Peter Christen
b0095c8d3c flush the compressor cache when a cleanup is done 2012-06-07 19:42:33 +02:00
Michael Peter Christen
a61f44f9e4 lazy initialization of block rank table. 
this causes that the table is not initialized when there is no search is
done. the effect is most strong if YaCy is started headless which causes
no browser pop-up which otherwise would load the search page and
therefore trigger the initialization of the table.
 2012-06-07 13:16:38 +02:00
Michael Peter Christen
96e9d77270 Merge branch 'master' of ssh://git@gitorious.org/yacy/rc1.git 
Conflicts:
	source/net/yacy/cora/sorting/WeakPriorityBlockingQueue.java
 2012-06-06 20:13:28 +02:00
Michael Peter Christen
00f2df1120 a variety of possible memory leak fixes 2012-06-06 18:23:18 +02:00
Michael Peter Christen
3dd8376825 added automatic cleaning of cache if metadata and file database size is 
not equal. It might happen that these data is different because one of
that caches is cleaned after a while or when it is too big. The metadata
is then not cleaned, but now wiped after a checkup process at every
application start. This should cause a bit less memory usage.
 2012-06-06 14:15:24 +02:00
Michael Peter Christen
d0ec8018f5 fixes for bad long computation 2012-06-06 14:13:31 +02:00
Michael Peter Christen
6bb07afcc3 accept also files with other file prefix; used to read 'foreign' cache 
files
 2012-06-06 13:36:10 +02:00
Michael Peter Christen
96c8119b50 added GeoLocation / GeoPoint classes which uses less memory than 
Location/Coordinates and has initializers with correct order of lat,lon
coordinates
 2012-06-06 12:57:42 +02:00
Michael Peter Christen
461a0ce052 removed warnings 2012-06-05 20:03:43 +02:00
Michael Peter Christen
62ae9bbfda allow more POIs, get more at once 2012-06-05 18:29:54 +02:00
Michael Peter Christen
407fdf6968 more bug fixes and performance hacks for search process 2012-06-05 15:04:23 +02:00
Michael Peter Christen
a1fe65b115 performance hacks 2012-06-05 12:06:26 +02:00
Michael Peter Christen
2fe207f813 Merge branch 'master' of ssh://git@gitorious.org/yacy/rc1.git 2012-06-04 23:44:38 +02:00
Michael Peter Christen
5aee19daa4 added show from cache in search results (not yet finished) 2012-06-04 23:44:26 +02:00
Michael Peter Christen
5e562dcdb7 adopted vocabulary usage within anotation/naviagtion feature of search 
to new SimpleVocabulary class
 2012-06-04 23:43:30 +02:00
Michael Peter Christen
514700291a moved Vocabulary to cora package (added in git 
964406ad17)
 2012-06-04 23:41:36 +02:00
Michael Peter Christen
0284a4d88f more fixes for double precision of coordinates 2012-06-04 23:37:41 +02:00
Michael Peter Christen
964406ad17 added concurrency enhancement to xml parser 2012-06-04 23:35:56 +02:00