Marown/yacy_search_server
1
0
Fork 0
mirror of https://github.com/yacy/yacy_search_server.git synced 2024-09-21 00:00:13 +02:00
Code Issues Actions Packages Projects Releases Wiki Activity
8,805 Commits 1 Branch 20 Tags 836 MiB
a3d5959981
Commit Graph

268 Commits

Author SHA1 Message Date
Michael Peter Christen
b9dfca4b0a - fixed IndexFederated Servlet / a embedded Solr can now be selected 
- added code stub for an embedded Solr but generation of Solr store is
still commented out (it works but is not yet ready for usage)
 2012-06-25 11:34:38 +02:00
Michael Peter Christen
fad3b14813 added jetty libraries, needed for future use as web server and as 
application server for the solr search interface
 2012-06-22 15:31:17 +02:00
Michael Peter Christen
a38b0a2c46 extended embedded solr tests to ensure that it will be usable within a 
jetty instance
 2012-06-22 11:40:02 +02:00
Michael Peter Christen
b9d42fd9c8 using com.google.common.io.Files instead of homebrew methods 2012-06-22 11:39:17 +02:00
Michael Peter Christen
a5eb91fa60 refactoring 2012-06-22 00:49:32 +02:00
Michael Peter Christen
1be0025a9c - added test for EmbeddedSolrConnector 
- added needed libraries for this test
this includes most (all) files needed for an embedded solr
 2012-06-22 00:36:49 +02:00
Michael Peter Christen
e12bb254b4 Merge branch 'master' of ssh://git@gitorious.org/yacy/rc1.git 2012-06-21 14:55:50 +02:00
Michael Peter Christen
3f55dc7c1e - added solr core and libraries that solr needs (lucene is missing, will 
follow later)
- added embedded solr connector which can connect to solr
programmatically (without using a server in between)
 2012-06-21 14:55:38 +02:00
Michael Peter Christen
786be7d175 better integration of RDFaParser 2012-06-20 16:39:04 +02:00
Michael Peter Christen
0752983fbd - automatic periodic saving of triplestore 
- transaction-safe storage of triplestore
 2012-06-17 10:50:12 +02:00
Michael Peter Christen
9264d8b4af removed old navigation practice using subject tags in favor of 
triplestore-tags
 2012-06-17 00:33:40 +02:00
Michael Peter Christen
64c0268b2b show triplestore metadata in yacydoc and viewfile 2012-06-16 17:40:15 +02:00
cominch
a95127c9af Triplestore: initalize per-user triplestores 2012-06-14 11:46:53 +02:00
Michael Peter Christen
e89747bb67 - added automated generation of vocabularies from url stubs 
- added clear of all terms for vocabularies
- added deletion of vocabularies
 2012-06-13 15:53:18 +02:00
Michael Peter Christen
8b53771db2 changed behavior of navigation processing: 
- vocabulary annotation is not done any more into the metadata of urldb
- vocabularies are written into the jena triplestore using a rdf
vocabulary
- vocabularies for rdf tripel must be updated; refactoring done
- with the new navigation tags in the triplestore a faster
pre-urldb-lookup is possible: navigation is processed now within the RWI
during pre-ranking retrieval
- added also a Owl vocabulary stub to add the plain-text url to the
triplestore using the owl:sameas predicate
 2012-06-11 23:49:30 +02:00
Michael Peter Christen
5fc6524ca8 - moved triple store to net.yacy.cora.lod (should be generalized there 
later
- added abstract add, delete, get methods in the triplestore
- added generation of triples after auto-annotation
- migrated all MultiProtocolURI objects to DigestURI in the parser since
the url hash is needed as subject value in the triples in the triple
store
 2012-06-11 16:48:53 +02:00
Michael Peter Christen
4ee6fb1de9 added missing blacklist dht cache storage (maybe due to mistakes in 
cherry picking)
 2012-06-11 00:38:02 +02:00
Roland 'Quix0r' Haeder
edaa09b9b1 Rewrote all String blacklist types to enum 'BlacklistType', closes bug 
#143

Conflicts:
	htroot/Supporter.java
	htroot/yacy/crawlReceipt.java
	htroot/yacy/transferRWI.java
	htroot/yacy/transferURL.java
	source/de/anomic/crawler/CrawlStacker.java
	source/de/anomic/data/ListManager.java
	source/net/yacy/peers/Protocol.java
	source/net/yacy/repository/Blacklist.java
	source/net/yacy/repository/LoaderDispatcher.java
	source/net/yacy/search/Switchboard.java
	source/net/yacy/search/index/MetadataRepository.java
	source/net/yacy/search/index/Segment.java
	source/net/yacy/search/query/RWIProcess.java
	source/net/yacy/search/snippet/MediaSnippet.java
 2012-06-11 00:17:30 +02:00
Roland 'Quix0r' Haeder
af5a597e47 Scroogle is not comming back, remove dead code 
Conflicts:
	source/net/yacy/search/Switchboard.java
 2012-06-10 23:38:41 +02:00
cominch
65c5826d93 bugfix 
Conflicts:
	source/net/yacy/document/parser/augment/AugmentParser.java
 2012-06-10 13:11:54 +02:00
Michael Peter Christen
cde20911bb saved a bit more ram using UTF8 String compression for OpenGeoDB and 
Geonames data files.
 2012-06-09 10:07:11 +02:00
Michael Peter Christen
2280a7b276 - changed initialization order to prefer allocation of memory for table 
files first
- bugfixes in memory amount calculation
 2012-06-09 09:05:47 +02:00
Michael Peter Christen
0746308bc2 only the metadata tables shall be able to use the tail cache 2012-06-08 18:36:11 +02:00
Michael Peter Christen
41c02cb10e - less restrictions for usage of Table RAM copy 
- new limit to use the table copy (instead of flag): 400MB available. If
less is available, then a copy is never used. If more is available, then
it can be used if there is a remaining space of at least 200MB
- flush caches more often: flush the Digest cache
 2012-06-08 12:48:25 +02:00
Michael Peter Christen
dd14b19c26 lazy initialization of block rank table ... only normal web search uses 
this. When interactive search or location search is used, the block rank
is switched off
 2012-06-08 09:41:29 +02:00
Michael Peter Christen
701b9a28a0 Merge branch 'master' of ssh://git@gitorious.org/yacy/rc1.git 
Conflicts:
	htroot/PerformanceMemory_p.java
 2012-06-08 09:16:16 +02:00
Michael Peter Christen
ab7107b34b fixed RWIProcess queue limits: now discovering hidden results for mass 
result retrieval
 2012-06-08 09:14:54 +02:00
Michael Peter Christen
b0095c8d3c flush the compressor cache when a cleanup is done 2012-06-07 19:42:33 +02:00
Michael Peter Christen
a61f44f9e4 lazy initialization of block rank table. 
this causes that the table is not initialized when there is no search is
done. the effect is most strong if YaCy is started headless which causes
no browser pop-up which otherwise would load the search page and
therefore trigger the initialization of the table.
 2012-06-07 13:16:38 +02:00
Michael Peter Christen
96e9d77270 Merge branch 'master' of ssh://git@gitorious.org/yacy/rc1.git 
Conflicts:
	source/net/yacy/cora/sorting/WeakPriorityBlockingQueue.java
 2012-06-06 20:13:28 +02:00
Michael Peter Christen
00f2df1120 a variety of possible memory leak fixes 2012-06-06 18:23:18 +02:00
Michael Peter Christen
d0ec8018f5 fixes for bad long computation 2012-06-06 14:13:31 +02:00
Michael Peter Christen
461a0ce052 removed warnings 2012-06-05 20:03:43 +02:00
Michael Peter Christen
407fdf6968 more bug fixes and performance hacks for search process 2012-06-05 15:04:23 +02:00
Michael Peter Christen
a1fe65b115 performance hacks 2012-06-05 12:06:26 +02:00
Michael Peter Christen
2fe207f813 Merge branch 'master' of ssh://git@gitorious.org/yacy/rc1.git 2012-06-04 23:44:38 +02:00
Michael Peter Christen
5e562dcdb7 adopted vocabulary usage within anotation/naviagtion feature of search 
to new SimpleVocabulary class
 2012-06-04 23:43:30 +02:00
Michael Peter Christen
240045cf7c fix for bad distance computation 2012-06-04 16:33:16 +02:00
Michael Peter Christen
e0d8643226 - performance hacks 
- added log warnings in case that search processes run into time-out
situations
- better concurrency for Integer formatter (used a non-synchronized
formatter before)
- bugfix for search termination (a poison pill was missing)
- added timeout parameters for search (again) -> target is, that they
are never reached.
 2012-06-04 15:37:39 +02:00
Michael Peter Christen
9b4c699526 ehanced location search: 
- search request are now made using a map boundary
- search results are only computed for the map boundary
- the number of results is adopted to the results in the visible range
- added a double-buffering for the search result markers
- added a search query option for the search results:
/radius/<lat>/<lon>/<radius>
 2012-05-31 22:39:53 +02:00
Michael Peter Christen
834dc6b263 store more data from interface access 2012-05-31 00:47:07 +02:00
Michael Peter Christen
10da7335ea performance hack: use a hash cache for all hashes that are computed by a 
byte array. If this hash is used in a HashMap (which is very often the
case) then this hack eliminates a lot of re-computations of the same
hash.
 2012-05-30 16:59:13 +02:00
Michael Peter Christen
7c1feefb28 introduced a default 10 second time-out in rwi normalization time 
uring search process to prevent endless deadlocks after a very long
running search
 2012-05-30 16:26:05 +02:00
Michael Peter Christen
c846e9ca14 redesign of the crawler monitor page: show crawled pages instead of 
queue of urls that shall be crawled
 2012-05-25 01:45:38 +02:00
Michael Peter Christen
c15fcde1c8 add-on to latest commit 2012-05-21 17:52:30 +02:00
Michael Peter Christen
cf47d94888 performance hack to parse numbers inside of substrings without actually 
generating a substring. This avoids the allocation of a String object
ech time a substring is parsed. Should affect CPU load during RWI
transmission.
 2012-05-21 13:40:46 +02:00
Michael Peter Christen
7e0ddbd275 added a "fromCache" flag in Response object to omit one cache.has() 
check during snippet generation. This should cause less blockings
 2012-05-21 03:03:47 +02:00
Michael Peter Christen
7bf421b9dd - fixed image search page navigation 
- removed some deadlocks and ConcurrentModificationExceptions during
DidYouMean collection
 2012-05-21 01:58:29 +02:00
Michael Peter Christen
fb94b47b1a changed queue sizes to have less memory occupied during indexing 2012-05-21 00:19:03 +02:00
Michael Peter Christen
76157dc2c3 bugfix for http://bugs.yacy.net/view.php?id=173 2012-05-21 00:18:00 +02:00
Michael Peter Christen
c6558cba08 more classification bugs 2012-05-20 02:59:47 +02:00
Michael Peter Christen
082831b9d6 search contentdom was checked in wrong way - fixed 2012-05-20 01:23:02 +02:00
reger
ee553d971e correct typo in scripts_txt comment 2012-05-19 02:09:16 +02:00
Michael Peter Christen
f294f2e295 bugfix to http://bugs.yacy.net/view.php?id=181 
tried to make a bit less 'noise' to dns server

also included: less processes in snippet fetch to reduce load during
search on small computers
 2012-05-19 01:06:33 +02:00
Michael Peter Christen
acf8d521a2 fix for http://bugs.yacy.net/view.php?id=126 2012-05-19 00:21:03 +02:00
Michael Peter Christen
bb88878b4d the last commit was incomplete.. 2012-05-18 22:33:16 +02:00
Michael Peter Christen
d320a31ae1 bugfix for http://bugs.yacy.net/view.php?id=186 2012-05-18 22:18:47 +02:00
Michael Peter Christen
3e1bc9477f Merge branch 'master' of ssh://git@gitorious.org/yacy/rc1.git 2012-05-17 13:58:09 +02:00
Roland 'Quix0r' Haeder
d10627d591 More sync in close() methods 
Conflicts:
	source/net/yacy/kelondro/logging/GuiHandler.java
	source/net/yacy/kelondro/workflow/InstantBusyThread.java
 2012-05-17 06:03:18 +02:00
Roland 'Quix0r' Haeder
b3ae2aa41f With or without 'final'? At least please try it in other methods 
Conflicts:
	source/de/anomic/tools/tarTools.java
 2012-05-17 06:00:49 +02:00
Roland 'Quix0r' Haeder
fbb946f913 Made a method static (Eclipse suggested it), removed unused import, pk=null check does now output a warning in logfile 2012-05-17 05:55:44 +02:00
Michael Peter Christen
52d307c735 prevent that the snippet fectch process removes catchall entries 2012-05-17 05:18:52 +02:00
Michael Peter Christen
89142d1e8d removed (not all) warnings 2012-05-16 13:42:32 +02:00
Michael Peter Christen
5deebd02ea added serialization 2012-05-15 23:10:47 +02:00
reger
b2175ea4ef Add possibility to set custom Solr field names for the YaCy default Solr attributes. 
- Changing the format of YaCy's solr.key.list while maintainig backward compatibility
  Federated index config screens adjusted accordingly
- modified the Solr update request to use a 3 min Solr autocommit intervall
 2012-05-15 22:34:02 +02:00
Michael Peter Christen
e7e381d110 added configuration to switch off redirection following in crawler 2012-05-15 12:25:46 +02:00
Michael Peter Christen
2717c1b749 fixed bug in solr interface 2012-05-15 12:25:14 +02:00
Michael Peter Christen
f150bc218b fixed bug in solr error document 2012-05-14 14:56:21 +02:00
Michael Peter Christen
cb54c1737b solrj connector bugfix 2012-05-14 11:56:04 +02:00
Roland 'Quix0r' Haeder
a093ccf5eb Now used synchronization in all close() methods to make sure all objects 
are 'closed' in an ordered way

Conflicts:
	source/de/anomic/http/server/ChunkedInputStream.java
	source/de/anomic/http/server/ChunkedOutputStream.java
	source/de/anomic/http/server/ContentLengthInputStream.java
	source/net/yacy/cora/protocol/Domains.java
	source/net/yacy/cora/services/federated/solr/SolrShardingConnector.java
	source/net/yacy/cora/services/federated/solr/SolrSingleConnector.java
	source/net/yacy/document/content/dao/PhpBB3Dao.java
	source/net/yacy/document/parser/html/AbstractTransformer.java
	source/net/yacy/kelondro/blob/BEncodedHeap.java
	source/net/yacy/kelondro/blob/HeapReader.java
	source/net/yacy/kelondro/index/RAMIndexCluster.java
	source/net/yacy/kelondro/io/ByteCountInputStream.java
	source/net/yacy/kelondro/logging/ConsoleOutErrHandler.java
	source/net/yacy/kelondro/table/SQLTable.java
 2012-05-14 07:41:55 +02:00
Michael Peter Christen
0d58fea210 made multiple connector default 2012-05-12 10:39:01 +02:00
Michael Peter Christen
adeb33bb36 better abstraction for solr objects 2012-05-09 17:21:19 +02:00
Michael Peter Christen
8864141872 more abstraction in solr connection classes 2012-05-09 17:00:56 +02:00
Michael Peter Christen
c00efc2717 made the solr connection more generic 2012-05-09 16:46:45 +02:00
Michael Peter Christen
ea2bd43b28 patch for broken configurations 2012-05-09 12:29:07 +02:00
Michael Peter Christen
ba6aaabc51 refactoring + parser bugfixes 2012-05-04 17:28:27 +02:00
Michael Peter Christen
453010bd68 - solved problems with backpath normalization 
- redesigned in/outbound link handover
- removed iframe links from inbound/outbound in solr scheme
 2012-04-27 16:48:51 +02:00
Michael Peter Christen
5f5ed33ed8 patch for media search (audio, video apps) 2012-04-27 14:18:02 +02:00
Michael Peter Christen
19efbf1b0f - apply directDocByURL to NOLOAD Queue 
- choose pushing to NOLOAD as default for site crawl
 2012-04-26 00:23:18 +02:00
Michael Peter Christen
659178942f - Redesigned crawler and parser to accept embedded links from the NOLOAD 
queue and not from virtual documents generated by the parser.
- The parser now generates nice description texts for NOLOAD entries
which shall make it possible to find media content using the search
index and not using the media prefetch algorithm during search (which
was costly)
- Removed the media-search prefetch process from image search
 2012-04-24 16:07:03 +02:00
Michael Peter Christen
a3badd3205 changed search process for images: no more media snippet load process, 
show only links from index which had been on the text search page
before. This creates a superfast search process for images!
 2012-04-24 12:55:58 +02:00
Michael Peter Christen
f8cd57c92f new indexing strategy: ALL links that appear anywhere are indexed, not 
only links where the content can be parsed. All non-parseable links are
placed into the noload queue. The search process must therefore be able
to filter out non-text search results.
- This fixes the problem that image search results appeared in the text
search.
- The interactive search can retrieve now ALL types of links
- The p2p interface is now extended to retrieve only certain types of
links (text, image, video, apps)
- The search process has an extension to filter the right document type
according to the search query
 2012-04-22 02:05:17 +02:00
Michael Peter Christen
14f67f217c refactoring of ContentDomain: now subclass of Classification 2012-04-22 00:04:36 +02:00
Michael Peter Christen
a1a5b015d8 refactoring: moved document Classification to cora package 2012-04-21 21:31:13 +02:00
Michael Peter Christen
33d1062c79 refactoring: the cache belongs to the crawler 2012-04-21 13:34:07 +02:00
Michael Peter Christen
7b5b9baee0 added citation rank to ranking profile 2012-04-16 23:43:50 +02:00
Michael Christen
02e4dedff2 fix to url citation collection 2012-04-13 11:52:59 +02:00
Michael Christen
e32055aa15 added stub classes for 
- a new database for url reference data ('seen links')
- a new database extending the references to the full url metadata
attributes set which shall replace the old metadata database if it is
finished
- migration help classes stub to use old and new metadata databases
simultanously
 2012-04-13 07:09:15 +02:00
Michael Christen
ac5d124ee0 experimental implementation of a citation ranking as post-ranking 
method. (ranking coefficient fixed, need to be made configurable)
 2012-04-13 06:47:33 +02:00
Michael Christen
8fc86fe397 added storage of full anchor link structure: 
the links between all pages are now stored. The same index structure as
used for the word index is used to make a reverse link index.
The new file(s) in SEGMENT/default/citation.index.*.blob store the
citation index. This will be used to create much more detailed link
structures for the YaCy apis and to create a better ranking. A ranking
using the citation.index should provide better results especially for
portal indexes and initranets.
 2012-03-29 17:20:14 +02:00
Lotus
0b3f39136e allow custom ppm lower than minimum button on /Crawler_p.html 
fixes http://bugs.yacy.net/view.php?id=166
 2012-03-17 20:43:19 +01:00
Michael Peter Christen
8aba045ba1 if a new pop-up page is set in config portal, then this page applies 
also to the default page configuration for the httpd if no path is
given.
 2012-02-26 20:53:32 +01:00
Michael Peter Christen
36e4d82b27 changed ranking 2012-02-25 12:58:12 +01:00
Michael Peter Christen
096c17e7cd added test code 2012-02-25 12:42:13 +01:00
Michael Peter Christen
9ad1d8dde2 complete redesign of crawl queue monitoring: do not look at a 
ready-prepared crawl list but at the stacks of the domains that are
stored for balanced crawling. This affects also the balancer since that
does not need to prepare the pre-selected crawl list for monitoring. As
a effect:
- it is no more possible to see the correct order of next to-be-crawled
links, since that depends on the actual state of the balancer stack the
next time another url is requested for loading
- the balancer works better since the next url can be selected according
to the current situation and not according to a pre-selected order.
 2012-02-02 21:33:42 +01:00
Michael Peter Christen
e2f8f263e8 changed storage of search words: keep order 2012-02-01 18:13:31 +01:00
Michael Peter Christen
2e5cd6a1b2 fixed parser extension deny list generation and usage 2012-02-01 00:15:59 +01:00
Michael Peter Christen
3cd6dcd352 do not add new solr fields as activated fields 2012-01-31 22:21:48 +01:00
Michael Peter Christen
e3bb73c3d6 serialized some database access methods 2012-01-31 21:13:49 +01:00
Michael Peter Christen
355ecf330f reduced target file site to 64mb 2012-01-29 20:35:48 +01:00
Michael Peter Christen
2ea585d616 fix for host navigator 2012-01-26 18:10:34 +01:00
Michael Peter Christen
4c5edab1ec added option to have exception search result windows 2012-01-26 15:32:30 +01:00
Michael Peter Christen
ef78f22ee1 performance hack 2012-01-25 12:48:48 +01:00
Michael Peter Christen
41536eb4a2 performance hack 2012-01-25 12:28:56 +01:00
Michael Peter Christen
f91487fc50 added delete-button for host navigation 2012-01-25 11:19:18 +01:00
Michael Peter Christen
e8d24fd802 author navigator can be switched off 2012-01-25 11:11:42 +01:00
Michael Peter Christen
558ab7bd4e made the protocol navigator reversible 2012-01-25 02:54:52 +01:00
Michael Peter Christen
96cb75f1d4 made the filetype navigator be able to deselect the search constraint 2012-01-25 02:50:06 +01:00
Lotus
c73af39e54 refactoring of tray icon class, 
now uses Java 6 methods natively
 2012-01-18 20:47:09 +01:00
Michael Peter Christen
4eff0e26f1 npe bugfix 2012-01-17 23:39:57 +01:00
Michael Peter Christen
1a0b6b3913 get more navigation details to search results 2012-01-17 16:44:30 +01:00
Michael Peter Christen
83009d86f7 added the vocabulary navigator. It can be very simply tested by 
switching on the locale dictionaries.
 2012-01-17 01:53:08 +01:00
Michael Peter Christen
254adea51c small fixes 2012-01-13 11:24:08 +01:00
Michael Peter Christen
c602eaaf46 enhanced search process 2012-01-10 03:00:55 +01:00
Michael Christen
eff966f396 fix for search process (it was aborted too early during remote search) 2012-01-09 03:02:35 +01:00
Marek Otahal
72adbeae90 !Important: move from Hashtable to HashMap 
Hashtable is an obsolete collection v1, now since v2 offers HashMap with same or better
functionality. Please review, almost all code was already moved, so only a few changes. That is not the issue,
but I found notices that some (ugly big) helper classes had to be created in past
to compensate missing Hashtable's functionality. I'd like input if we can remove some of them.
look for //FIX: if these commits

Signed-off-by: Marek Otahal <markotahal@gmail.com>
 2012-01-09 01:29:18 +01:00
Marek Otahal
f40efb39af Blacklist loadList() remove duplicates by using Set 
Signed-off-by: Marek Otahal <markotahal@gmail.com>
 2012-01-09 01:18:01 +01:00
Michael Peter Christen
2ee8cbeb2c Merge branch 'master' of ssh://git@gitorious.org/yacy/rc1.git 
Conflicts:
	source/net/yacy/search/Switchboard.java
 2012-01-05 18:37:46 +01:00
Michael Peter Christen
992dbdf4bb added noload statistic to servlets 2012-01-05 18:33:05 +01:00
Michael Christen
216a287a85 Merge commit '6d4e08ed06c5cd28c45981b2ebe31c7f7ec6fd83' into quix0r 
Conflicts:
	source/de/anomic/crawler/CrawlQueues.java
 2012-01-04 20:16:37 +01:00
stbrumm
d18095dc48 Patch fuer Issue 0000102 
and fixes to Patch (private peer status is a property of a peer, not a
status)
 2012-01-03 17:49:37 +01:00
Michael Christen
585a8f3c44 fixed a bug in search sequence (caused emtpy results) 2012-01-02 02:10:39 +01:00
Roland 'Quix0r' Haeder
a3083d13bf Blacklist checks are now always turned on, in media searches (e.g. image search) images matching blacklist entries are no longer shown to the user 2011-12-28 20:09:17 +01:00
Michael Christen
52184a1170 fix for search process 2011-12-27 23:43:44 +01:00
Michael Christen
0797b0de99 new handling of remote search processes: looking for seeds will now not 
block the whole search process any more. A deadlock with a DHT selection
process may have been the cause for interface lockings in the past.
 2011-12-21 00:32:03 +01:00
Michael Christen
9e5894c784 Removed handling of components objects for URIMetadataRows. 
This is a preparation to replace this rows with nodes from the node
store.
 2011-12-17 01:27:08 +01:00
Michael Christen
c04bfaa51b refactoring 2011-12-16 23:59:29 +01:00
Michael Christen
e9dc99fe15 added rules to set specific RWIs as private RWIs which are not 
transmitted to remote peers. This will be used for private index copies
and phonetic indexes.
 2011-12-14 22:15:51 +01:00
Michael Peter Christen
0bcef2d156 added feature as requested in 
http://forum.yacy-websuche.de/viewtopic.php?f=18&t=3461
The search can now be configured with a non-display host list.
the search will always exlude the given list of host unless they are
requested directly using the host navigation
 2011-12-13 00:16:05 +01:00
Michael Christen
3eccdca63c protection against too long running snippet fetch processes 2011-12-07 00:32:42 +01:00
Michael Christen
86b3385847 fixed a deadlock during secondary remote search 2011-12-07 00:18:34 +01:00
Michael Christen
c715d19c09 fixes for dependency on svn 2011-12-06 22:05:22 +01:00
Michael Christen
0bc5d76bee ups 2011-12-06 21:55:50 +01:00
Michael Christen
044f83feed added some pauses into the search process which shall produce 
better-ranked search results. without that pauses the result page will
only contain links from the peer that answers first which is not a good
average picture of all the peers that provided results
 2011-12-06 15:28:48 +01:00
Michael Christen
f14faf503b better ranking because we wait a very little time during the search 
process more to get better remote sear results into the ranking priority
stack
 2011-12-06 02:24:51 +01:00
orbiter
f9216e388c - faster ping to clean up old peers faster 
- clean up more news

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@8125 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2011-11-30 21:21:16 +00:00
orbiter
d9c066227a fix for npe 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@8122 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2011-11-28 13:36:51 +00:00
orbiter
ebd840ebf6 - enhanced description on search front page 
- fixed language and heuristic modifier
- added hint to crawl start that we can do also ftp and smb crawls
- added a protocol extension to remote crawls to transport all search modifiers to remote peers

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@8108 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2011-11-26 13:40:33 +00:00
orbiter
e22f8497c9 - tested the ARC methods 
- removed strict authentication (if password is empty; this was buggy and not useful; can be switched on if necessary globally and not for each interface method)
- increased speed of CrawlResults page (no dns lookup any more)
- increased speed of favicon display (removed dns lookup)

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@8104 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2011-11-25 14:09:25 +00:00
orbiter
bc5df0eef5 updated ranking tables (fresh computation) 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@8103 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2011-11-25 12:37:00 +00:00
orbiter
5a55397f99 some last-minute performance hacks 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@8101 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2011-11-25 11:23:52 +00:00
orbiter
c9216d5adf fixed secondary remote search (the process that finds distributed join situations) 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@8098 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2011-11-24 22:45:31 +00:00
orbiter
64fd20b857 new default ranking profile 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@8097 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2011-11-24 20:07:52 +00:00
orbiter
0cf9ebc3b0 speed enhancements when parsing RWI rows (makes search slightly faster) 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@8096 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2011-11-24 17:37:04 +00:00
orbiter
ee8b1d4de1 fixed unresolved pattern and unwanted local/global switch when using votes on search results 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@8093 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2011-11-24 16:05:09 +00:00
orbiter
c584db991f creating a bookmark from the search results now works again .. with new YMarks 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@8092 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2011-11-24 14:57:09 +00:00
orbiter
6cd27473f5 - better default values for caching and cache usage 
- set new caching and verification behavior according to use case automatically

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@8087 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2011-11-24 10:22:02 +00:00
orbiter
1019c36dad bug fixes and speed enhancements for search 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@8085 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2011-11-24 01:30:12 +00:00
orbiter
507c9d478d much better timing when search globally; less blocking; more results earlier! 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@8084 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2011-11-23 23:39:34 +00:00
orbiter
8e0b2c5832 fixed cluster search 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@8083 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2011-11-23 22:21:14 +00:00