Marown/yacy_search_server
1
0
Fork 0
mirror of https://github.com/yacy/yacy_search_server.git synced 2024-09-21 00:00:13 +02:00
Code Issues Actions Packages Projects Releases Wiki Activity
8,876 Commits 1 Branch 20 Tags 836 MiB
00c1c777fa
Commit Graph

189 Commits

Author SHA1 Message Date
Michael Peter Christen
00c1c777fa refactoring 2012-09-21 15:48:16 +02:00
orbiter
563d584420 removed more dependencies in cora from kelondro 2012-09-21 11:02:36 +02:00
Michael Peter Christen
e65cecc419 - updated lucene libraries to 3.6.1 
- added lucene-grouping which enables faceted search; try this:
http://localhost:8090/solr/select?q=*:*&start=0&rows=3&facet=true&facet.field=host_s
 2012-09-10 10:12:38 +02:00
Michael Peter Christen
4d29f59a27 removed warnings 2012-09-10 07:15:52 +02:00
apfelmaennchen
d31a632951 - added dmoz RDF dump importer 
- added indexing to Tables columns to support larger bookmark
collections
- added RDF output (HTTP) for public bookmarks at /YMarks.rdf
- YMarkRDF also provides a Jena RDF Model as "internal" API
- various other changes/fixes for YMarks (mainly backend)
 2012-09-09 09:53:58 +02:00
orbiter
2094df2e4e - correct length computation for BStringObject (bugfix suggested by 
apfelmaennchen)
- using ASCII for string conversion for Strings generated from Integer
 2012-08-26 17:46:40 +02:00
Michael Peter Christen
94a334f128 another fix to the Solr metadata reading process and to the shutdown 
process
 2012-08-13 11:13:53 +02:00
sixcooler
f32aa9a49c prevent merge of blobs that can't be handled in memory 2012-07-31 23:23:16 +02:00
Michael Peter Christen
1687737771 Abstraction of HandleMap and HandleSet 2012-07-27 12:13:53 +02:00
Michael Peter Christen
e432bb9cd9 better calculation of possible saving in HeapReader index data structure 2012-07-26 10:05:06 +02:00
Michael Peter Christen
9549984c65 documentation/comments 2012-07-25 21:34:23 +02:00
Michael Peter Christen
f78ce93a80 collection of speed and memory saving hacks 2012-07-13 21:15:38 +02:00
orbiter
482afed07c reduced logging overhead (a bit) 2012-07-12 19:23:40 +02:00
orbiter
0cbda0b2b8 - replaced all length() == 0 and size() == 0 with isEmpty() 
- replaced some length() > 0 and size() > 0 with !isEmpty() - cannot be
done automatically
- implemented some isEmpty() methods
 2012-07-10 22:59:03 +02:00
Michael Peter Christen
b0c408788b made class methods static where possible 2012-07-05 12:38:41 +02:00
Michael Peter Christen
0301aba1e9 removed unused method parameters 2012-07-05 10:23:07 +02:00
Michael Peter Christen
ea10766bfd cleaned unnecessary nested code 2012-07-05 08:44:39 +02:00
Michael Peter Christen
8a82609360 - smaller caches to save memory 
- close cloneable iterators to free memory
 2012-07-02 15:40:40 +02:00
Michael Peter Christen
0c345d1559 giving threads name so its easier to see whats happening during 
debugging and within a thread dump
 2012-07-02 09:51:43 +02:00
Michael Peter Christen
de3ef8ad73 removed unimportant warnings 2012-06-19 08:45:34 +02:00
Michael Peter Christen
bef823c247 close the reader if finished 2012-06-11 01:20:54 +02:00
cominch
9cbfc1a1c0 augmentedProxy, which forwards every proxy request to a 
rewrite engine to customize existing webpages. originally implemented by
Florian Richter.

Conflicts:
	source/de/anomic/http/server/HTTPDProxyHandler.java
 2012-06-10 10:15:34 +02:00
Michael Peter Christen
ba10caf89a lazy initialization of database tables 2012-06-08 09:30:51 +02:00
Michael Peter Christen
701b9a28a0 Merge branch 'master' of ssh://git@gitorious.org/yacy/rc1.git 
Conflicts:
	htroot/PerformanceMemory_p.java
 2012-06-08 09:16:16 +02:00
Michael Peter Christen
10c9c17d51 fixed handlemap spread factor and null iterator handling 2012-06-08 09:13:41 +02:00
Michael Peter Christen
b0095c8d3c flush the compressor cache when a cleanup is done 2012-06-07 19:42:33 +02:00
Michael Peter Christen
96e9d77270 Merge branch 'master' of ssh://git@gitorious.org/yacy/rc1.git 
Conflicts:
	source/net/yacy/cora/sorting/WeakPriorityBlockingQueue.java
 2012-06-06 20:13:28 +02:00
Michael Peter Christen
3dd8376825 added automatic cleaning of cache if metadata and file database size is 
not equal. It might happen that these data is different because one of
that caches is cleaned after a while or when it is too big. The metadata
is then not cleaned, but now wiped after a checkup process at every
application start. This should cause a bit less memory usage.
 2012-06-06 14:15:24 +02:00
Michael Peter Christen
6bb07afcc3 accept also files with other file prefix; used to read 'foreign' cache 
files
 2012-06-06 13:36:10 +02:00
Michael Peter Christen
461a0ce052 removed warnings 2012-06-05 20:03:43 +02:00
reger
6696cb1313 bugfix: lookup of peernames no result for active peer in page IndexControlRWIs_p.html -> Transfer RWI to other Peer 
SeedDB.lookupByName searche for lowercase peerNames, while MapColumnIndex.getIndex uses peername as is in the keyset.
Changed the index init to insert lowercase peer names as key
 2012-05-20 05:25:16 +02:00
Roland 'Quix0r' Haeder
a093ccf5eb Now used synchronization in all close() methods to make sure all objects 
are 'closed' in an ordered way

Conflicts:
	source/de/anomic/http/server/ChunkedInputStream.java
	source/de/anomic/http/server/ChunkedOutputStream.java
	source/de/anomic/http/server/ContentLengthInputStream.java
	source/net/yacy/cora/protocol/Domains.java
	source/net/yacy/cora/services/federated/solr/SolrShardingConnector.java
	source/net/yacy/cora/services/federated/solr/SolrSingleConnector.java
	source/net/yacy/document/content/dao/PhpBB3Dao.java
	source/net/yacy/document/parser/html/AbstractTransformer.java
	source/net/yacy/kelondro/blob/BEncodedHeap.java
	source/net/yacy/kelondro/blob/HeapReader.java
	source/net/yacy/kelondro/index/RAMIndexCluster.java
	source/net/yacy/kelondro/io/ByteCountInputStream.java
	source/net/yacy/kelondro/logging/ConsoleOutErrHandler.java
	source/net/yacy/kelondro/table/SQLTable.java
 2012-05-14 07:41:55 +02:00
Michael Peter Christen
ba6aaabc51 refactoring + parser bugfixes 2012-05-04 17:28:27 +02:00
Michael Peter Christen
213c8d97f2 use less proccesses in process pool 2012-02-25 14:07:20 +01:00
Michael Peter Christen
b4409cc803 small redesign of blob column index and usage 2012-02-02 06:43:57 +01:00
Michael Peter Christen
0b67a0a5d8 added a column index for tables in blob files. This is heavily used 
during receiving of DHT submissions and when answering remote search
requests. Both events together may have caused IO-deadlocking and this
commit shall fix that.
 2012-02-01 15:11:21 +01:00
Michael Peter Christen
37f2d1b3e9 replaced Thread initialization with ExecutorService pool for delete 
method. This is much faster and produces less blocking when using the
Compressor class which is used by the HTCache. I.e. picture search is
much faster now.
 2012-01-16 01:05:30 +01:00
Michael Peter Christen
0d6176804b emergency disabling of GenerationMemoryStrategy because of non-working 
available-method
 2012-01-15 21:58:18 +01:00
Michael Peter Christen
87f0210480 enriched log output to find NPE in HeapReader 2012-01-15 12:08:46 +01:00
Michael Christen
216a287a85 Merge commit '6d4e08ed06c5cd28c45981b2ebe31c7f7ec6fd83' into quix0r 
Conflicts:
	source/de/anomic/crawler/CrawlQueues.java
 2012-01-04 20:16:37 +01:00
Michael Christen
575dbbaa93 enhancements in Blob retrieval: try to use less CPU resources by testing 
a blog first that most certainly has wanted entries.
 2012-01-02 02:14:05 +01:00
Roland 'Quix0r' Haeder
fa08ed5ae5 Fixed a lot CHMOD rights (no need for execute flag on *.java/*.html) and introduced local/remote crawl size ratio based check 2011-12-29 00:33:16 +01:00
Michael Christen
c04bfaa51b refactoring 2011-12-16 23:59:29 +01:00
Michael Peter Christen
613ab6a69d added BEncodedHeapBag and BEncodedHeapShard which are storage container 
for a new metadata store. An abstraction of the content for this storage
is defined with MapStore. A MapStore is an abstraction of a RDF Node
store.
 2011-12-16 23:00:50 +01:00
Michael Christen
1f4afb4dc0 performance hacks 2011-12-15 15:15:53 +01:00
Michael Christen
e9dc99fe15 added rules to set specific RWIs as private RWIs which are not 
transmitted to remote peers. This will be used for private index copies
and phonetic indexes.
 2011-12-14 22:15:51 +01:00
Michael Christen
078fcde0dd bad initialization 2011-12-07 01:02:23 +01:00
Michael Christen
044f83feed added some pauses into the search process which shall produce 
better-ranked search results. without that pauses the result page will
only contain links from the peer that answers first which is not a good
average picture of all the peers that provided results
 2011-12-06 15:28:48 +01:00
Michael Christen
d35bdc2df6 removed npe 2011-12-05 23:37:49 +01:00
Michael Christen
9cd469e6d6 added pull request from als plus an NPE fix 2011-12-04 12:15:03 +01:00