Marown/yacy_search_server
1
0
Fork 0
mirror of https://github.com/yacy/yacy_search_server.git synced 2024-09-21 00:00:13 +02:00
Code Issues Actions Packages Projects Releases Wiki Activity
9,415 Commits 1 Branch 20 Tags 836 MiB
2c3b024196
Commit Graph

46 Commits

Author SHA1 Message Date
Michael Peter Christen
2d36a7eaf5 - do not create a new query for all remote peers 
- no document search this time
- adjusted banner and network to not show 'WORDS' but DHT Chunks. This
is to avoid confusion for robinson peers which do not create Word
Entries
 2013-03-15 00:14:28 +01:00
orbiter
d74472f562 corrected result counter 2013-02-27 22:40:23 +01:00
Michael Peter Christen
788288eb9e added the generation of 50 (!!) new solr field in the core 'webgraph'. 
The default schema uses only some of them and the resting search index
has now the following properties:
- webgraph size will have about 40 times as much entries as default
index
- the complete index size will increase and may be about the double size
of current amount
As testing showed, not much indexing performance is lost. The default
index will be smaller (moved fields out of it); thus searching
can be faster.
The new index will cause that some old parts in YaCy can be removed,
i.e. specialized webgraph data and the noload crawler. The new index
will make it possible to:
- search within link texts of linked but not indexed documents (about 20
times of document index in size!!)
- get a very detailed link graph
- enhance ranking using a complete link graph

To get the full access to the new index, the API to solr has now two
access points: one with attribute core=collection1 for the default
search index and core=webgraph to the new webgraph search index. This is
also avaiable for p2p operation but client access is not yet
implemented.
 2013-02-22 15:45:15 +01:00
orbiter
5dfd6359cb redesign of the QueryParams class: introduced QueryGoal which holds the 
query string parser. This shall be used to create a proper full-string
matching which is handled then by QueryGoal.
 2012-11-18 01:22:41 +01:00
Michael Peter Christen
8fb370d9f8 renovated the way how search results are count. should be correct now... 2012-11-05 03:19:28 +01:00
Michael Peter Christen
6629e37685 tried to clean up the search process mess 2012-11-01 17:16:43 +01:00
Michael Peter Christen
c5f67a5d6d fixed a problem with local search from solr results: now all results 
from solr are shown (again)
 2012-11-01 10:22:22 +01:00
sixcooler
2d972f289a rise commitWithinMs to default-value from SwitchBoard 
(result in lower hd-io)

no dots in memory-graph (there are to much of them)
 2012-10-26 02:12:45 +02:00
Michael Peter Christen
f2d0418218 because the new PngEncoder had a problem with the PixelGrabber which is 
caused by a JRE bug, the PixelGrabber had to be circumvented using an
own frame buffer which can be read without a PixelGrabber. This resulted
in ultra-fast and much less memory-consuming transformation. YaCy images
are now generated really fast!
 2012-10-25 17:59:20 +02:00
Michael Peter Christen
d5d64019e5 - added a method for the RasterPlotter to draw arrow endings to lines 
- replaced the dot in the NetworkGraph with arrows
- enhanced the image drawing speed using pre-computed color values
- added more attention for OOM cases during very large image painting
 2012-10-25 16:05:04 +02:00
Michael Peter Christen
ae6feb5610 showing the web structure graph as animation in the crawl monitor 2012-10-23 02:50:26 +02:00
Michael Peter Christen
39317a6c66 enhanced webstructure image: introduced 
- multiple hosts can be listed (comma-separated) as host argument
- new 'bf'-attribut (branch factor): the maximum number of edges per
node
- the bf-value is computed automatically
- ordering of nodes when the graphic is drawed: mostly the drawing ends
with an limitation eg. number of nodes. When this happens, it should be
ensured that more 'interesting' nodes are painted in advance. This is
now done by sorting all nodes by the number of links they have in de
distant sub-graph.
 2012-10-22 16:23:39 +02:00
Michael Peter Christen
43f3345c90 - removed dependencies from URIMetadataRow and made direct access to 
URIMetadataNode which creates the opportunity to access Solr objects
directly and use their information richness
- lazy initialization of the URIMetadataNode object - should cause less
computation and memory usage during search.
- removed dead code
 2012-10-16 18:11:57 +02:00
Michael Peter Christen
21fe8339b4 - enhanced generation of url objects 
- enhanced computation of link structure graphics
- enhanced collection of data for link structures
 2012-10-15 13:17:13 +02:00
Michael Peter Christen
016ffa7434 increased strength of crawling waves in network image 2012-09-26 23:32:13 +02:00
Michael Peter Christen
1533bfd63b refactoring 2012-09-25 21:20:03 +02:00
Michael Peter Christen
5683162bd3 simplifications in DHT Distribution class and more documentation 2012-09-24 12:01:09 +02:00
Michael Peter Christen
e57bf2ca39 simplified DHT classes 2012-09-24 01:04:39 +02:00
Michael Peter Christen
8219a445f3 refactoring 2012-09-21 16:46:57 +02:00
Michael Peter Christen
00c1c777fa refactoring 2012-09-21 15:48:16 +02:00
orbiter
563d584420 removed more dependencies in cora from kelondro 2012-09-21 11:02:36 +02:00
Michael Peter Christen
a06123aec6 more abstraction and less parameter overhead for remote search 2012-08-20 01:29:15 +02:00
Michael Peter Christen
24d9db1613 snippet retrieval loading processes may use a smaller minimum load time 
value than crawling processes. This speeds up the search result
preparation dramatically.
 2012-07-30 10:38:23 +02:00
Michael Peter Christen
1687737771 Abstraction of HandleMap and HandleSet 2012-07-27 12:13:53 +02:00
orbiter
0cbda0b2b8 - replaced all length() == 0 and size() == 0 with isEmpty() 
- replaced some length() > 0 and size() > 0 with !isEmpty() - cannot be
done automatically
- implemented some isEmpty() methods
 2012-07-10 22:59:03 +02:00
orbiter
62202e2d71 refactoring of query attribute variable names for better consistency 
with (next) stored query words
 2012-07-09 11:14:50 +02:00
Michael Peter Christen
0301aba1e9 removed unused method parameters 2012-07-05 10:23:07 +02:00
Michael Peter Christen
1825f165b8 better integration of blacklist according to use case 2012-07-02 13:57:29 +02:00
Michael Peter Christen
3b992e6b00 using utf8 String compression in Webstructure database 2012-06-09 11:00:33 +02:00
Michael Peter Christen
f294f2e295 bugfix to http://bugs.yacy.net/view.php?id=181 
tried to make a bit less 'noise' to dns server

also included: less processes in snippet fetch to reduce load during
search on small computers
 2012-05-19 01:06:33 +02:00
Michael Peter Christen
89142d1e8d removed (not all) warnings 2012-05-16 13:42:32 +02:00
Michael Peter Christen
15db703808 added missing serialization to remove all warnings 2012-05-15 13:13:07 +02:00
Roland 'Quix0r' Haeder
a093ccf5eb Now used synchronization in all close() methods to make sure all objects 
are 'closed' in an ordered way

Conflicts:
	source/de/anomic/http/server/ChunkedInputStream.java
	source/de/anomic/http/server/ChunkedOutputStream.java
	source/de/anomic/http/server/ContentLengthInputStream.java
	source/net/yacy/cora/protocol/Domains.java
	source/net/yacy/cora/services/federated/solr/SolrShardingConnector.java
	source/net/yacy/cora/services/federated/solr/SolrSingleConnector.java
	source/net/yacy/document/content/dao/PhpBB3Dao.java
	source/net/yacy/document/parser/html/AbstractTransformer.java
	source/net/yacy/kelondro/blob/BEncodedHeap.java
	source/net/yacy/kelondro/blob/HeapReader.java
	source/net/yacy/kelondro/index/RAMIndexCluster.java
	source/net/yacy/kelondro/io/ByteCountInputStream.java
	source/net/yacy/kelondro/logging/ConsoleOutErrHandler.java
	source/net/yacy/kelondro/table/SQLTable.java
 2012-05-14 07:41:55 +02:00
Marc Nause
a691023d04 *) better formatting for network QPM 
*) refactoring
 2012-05-08 20:07:34 +02:00
Michael Peter Christen
ba6aaabc51 refactoring + parser bugfixes 2012-05-04 17:28:27 +02:00
Michael Peter Christen
33d1062c79 refactoring: the cache belongs to the crawler 2012-04-21 13:34:07 +02:00
Michael Peter Christen
8c06925984 animation of the web structure picture 2012-02-25 15:42:29 +01:00
Michael Christen
0797b0de99 new handling of remote search processes: looking for seeds will now not 
block the whole search process any more. A deadlock with a DHT selection
process may have been the cause for interface lockings in the past.
 2011-12-21 00:32:03 +01:00
Michael Christen
044f83feed added some pauses into the search process which shall produce 
better-ranked search results. without that pauses the result page will
only contain links from the peer that answers first which is not a good
average picture of all the peers that provided results
 2011-12-06 15:28:48 +01:00
Michael Christen
e7e429705a - less automatic indexing after a search (needs to reset the default 
crawl profiles)
- fix for concurrency problem in storage of serverSwitch Properties
- markup update
 2011-12-05 16:22:11 +01:00
orbiter
4b8ff84705 - search bugfixes (page counter and number of results per page; recognition of new search) 
- experiments to speed-up the network image production (commented out)


git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@8130 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2011-12-02 09:16:24 +00:00
orbiter
0796b54601 - some speed hacks for network image 
- panic patch for 'AD' hashes until it is clear where the problem comes from

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@8126 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2011-11-30 22:06:13 +00:00
orbiter
35a9e8f307 - fixed network graphic 
- debuged evaluation tables
- changed cache settings in template engine
- some speed hacks
- changed int angles for peer positions in network graphic to double angles

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@8124 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2011-11-30 20:50:41 +00:00
orbiter
1120f0c93c update to network graphics: slightly less crawling activity, slightly stronger color for query activity 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@8089 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2011-11-24 12:55:09 +00:00
orbiter
a7df70221e refactoring 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7987 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2011-10-04 09:06:24 +00:00
orbiter
d2ea250d99 refactoring: 
- moved many classes from de.anomic to net.yacy
- made more sub-packages for search classes

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7973 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2011-09-25 16:59:06 +00:00