Marown/yacy_search_server
1
0
Fork 0
mirror of https://github.com/yacy/yacy_search_server.git synced 2024-09-19 00:01:41 +02:00
Code Issues Actions Packages Projects Releases Wiki Activity
9,038 Commits 1 Branch 20 Tags 836 MiB
9330ad4838
Commit Graph

5975 Commits

Author SHA1 Message Date
Michael Peter Christen
9330ad4838 - fixed the delete option in host browser 
- added a delete method which can be used to delete a full subpath in
solr.
 2012-11-02 01:22:31 +01:00
Michael Peter Christen
a63179f3f9 added the MIME attribute for the R tag in GSA search result writer 2012-11-02 00:14:29 +01:00
Michael Peter Christen
1168d09de8 more refactoring - integrated the code of SnippetProcess into 
SearchEvent
 2012-11-01 17:40:06 +01:00
Michael Peter Christen
6629e37685 tried to clean up the search process mess 2012-11-01 17:16:43 +01:00
Michael Peter Christen
c5f67a5d6d fixed a problem with local search from solr results: now all results 
from solr are shown (again)
 2012-11-01 10:22:22 +01:00
Michael Peter Christen
f8f05ecba7 - added a delete button in host browser to delete a complete subpath 
- removed storage of default collection name - default is now "user"
- made stacking of crawl start points concurrently
 2012-10-31 17:44:45 +01:00
Michael Peter Christen
0716a24737 added more / all new crawl profile fields into crawl profile editor 2012-10-31 15:13:05 +01:00
Michael Peter Christen
4a14122ba7 in case that a crawl profile has a collection assigned, use the 
collection to show a name in the web interface. This should prevent that
much too long names make the interface unusable.
 2012-10-31 14:08:33 +01:00
Michael Peter Christen
0fe8be7981 enhaced data structures for balancer and latency computation which 
should produce a bit better prognosis about forced waiting times.
 2012-10-30 17:30:24 +01:00
Michael Peter Christen
ac9540dfb6 removed options for stopwords which are not used 2012-10-30 12:36:36 +01:00
Michael Peter Christen
ce3fed8882 added the Google Search Appliance (GSA) api interface to the main menu. 
See:
https://developers.google.com/search-appliance/documentation/68/xml_reference#request_overview
 2012-10-30 12:27:22 +01:00
Michael Peter Christen
b2ffd49817 less latency 2012-10-30 12:26:32 +01:00
Michael Peter Christen
0833937c1c better balancing and duetime-cumputation also for no-delay intranet 
hosts
 2012-10-30 11:28:49 +01:00
Michael Peter Christen
c326aa8f67 disabled writing new entries to crawl stacks to prevent that a domain 
with many documents block refreshing of the crawl queue
 2012-10-29 22:26:52 +01:00
Michael Peter Christen
6905182d41 - fix for number of words log message 
- adding meta:refresh also to crawler stack
 2012-10-29 21:42:31 +01:00
Michael Peter Christen
c25d7bcb80 - added concurrency for robots.txt loading 
- changed data model for domain counter
 2012-10-29 21:08:45 +01:00
Michael Peter Christen
a94c537afc fixed getSize() which can use the cache size while the crawl is running 2012-10-29 11:56:07 +01:00
Michael Peter Christen
96912c9471 enhancement to solr caching: consider that during a get() the document 
is not in solr but the cache points out that a commit is needed to get
the document.
 2012-10-29 11:35:24 +01:00
Michael Peter Christen
a87811bc38 more auto-commit calls when a search interface is opened, but not when a 
search is done there to prevent blocking during search-time.
 2012-10-29 11:27:13 +01:00
Michael Peter Christen
3d3d654e88 if a network configuration is choosed which does not allow DHT and no 
P2P communication is in robinson mode) then some menu entries are
disabled which have no use in this mode.
 2012-10-29 01:51:19 +01:00
Michael Peter Christen
2d9e577ad0 replaced the custom robots.txt loader by the standard http loader 2012-10-28 22:48:11 +01:00
Michael Peter Christen
799d71bc67 enhanced solr caching: 
- increased cache size which is needed for longer solr commit time
- speed hacks on cache write code
 2012-10-28 20:31:29 +01:00
Michael Peter Christen
a33e2742cb - removed unnecessary synchronized and deadlock in crawler 
- removed problem with monitoring object on Balancer.wait
- added missing user agent settings
 2012-10-28 19:56:02 +01:00
orbiter
8952153ecf update to Balancer algorithm: 
- create a load list from the current list of known hosts
- do not create this list for each Balancer.pop access
- create the list from those hosts which have a zero-waiting time
- select 1/3 from that list which have the most urls waiting
- get hosts from the wainting list in random order
- fixes for some delta-time computations
- always load all urls from hosts which have never been loaded before
 2012-10-28 13:24:49 +01:00
orbiter
354f0d9acd moved static method from ClusteredScoreMap to MapDataMining because it 
was not used in the ClusteredScoreMap class but only in MapDataMining
 2012-10-28 11:29:53 +01:00
Michael Peter Christen
8e1248ffe3 force a commit in advance of a search for the administrator to get most 
recent results even if commit time is high and an indexing is ongoing.
 2012-10-26 15:35:42 +02:00
Michael Peter Christen
3b48c78190 added an option to force a commit to solr. 
may be used by a search front-end in case that the commitWithinMs time
is too short to get recently indexed documents.
 2012-10-26 07:39:07 +02:00
sixcooler
2d972f289a rise commitWithinMs to default-value from SwitchBoard 
(result in lower hd-io)

no dots in memory-graph (there are to much of them)
 2012-10-26 02:12:45 +02:00
orbiter
8fde1dd3b6 another performance and memory hack to graphics: this makes it possible 
to produce a 100-Megapixel png network graphic image on my 6 year old
laptop in standard configuration in 10 seconds.
 2012-10-25 21:40:27 +02:00
Michael Peter Christen
1baf498d59 - show more lines in online log 
- reverse order is default now
 2012-10-25 18:38:39 +02:00
Michael Peter Christen
55bdafbaf1 more image processing hacks 2012-10-25 18:20:05 +02:00
Michael Peter Christen
f2d0418218 because the new PngEncoder had a problem with the PixelGrabber which is 
caused by a JRE bug, the PixelGrabber had to be circumvented using an
own frame buffer which can be read without a PixelGrabber. This resulted
in ultra-fast and much less memory-consuming transformation. YaCy images
are now generated really fast!
 2012-10-25 17:59:20 +02:00
Michael Peter Christen
d5d64019e5 - added a method for the RasterPlotter to draw arrow endings to lines 
- replaced the dot in the NetworkGraph with arrows
- enhanced the image drawing speed using pre-computed color values
- added more attention for OOM cases during very large image painting
 2012-10-25 16:05:04 +02:00
Michael Peter Christen
85ca07b90e when a new crawl is started, an equal crawl, if still running, is 
terminated and the corresponding crawl profile is deleted (this also
clears the crawl queue entries for that crawl profile)
 2012-10-25 10:20:55 +02:00
Michael Peter Christen
906e51214a the web structure image shows the pivot dot in a different color 2012-10-25 10:18:28 +02:00
Michael Peter Christen
b3ffcde0c7 - prepared PngEncoder for concurrency: PixelGrabber.grabPixels is the 
main time-consuming process. This shall be done in concurrency.
- added concurrent processes to call the PixelGrabber and framework to
do that (queues)

It is now possible to create 4k-Images (3840x2160) i.e. with the Network
Graphics servlet
 2012-10-24 02:08:51 +02:00
Michael Peter Christen
e9c6f4ce2e - new order of data computation: first compute the size of 
compressed deflater output, then assign an exact-sized byte[] which
makes resizing afterwards superfluous
- after all enhancements all class objects were removed; result is just
one short static method
- made objects final where possible
 2012-10-24 00:41:09 +02:00
orbiter
c6a1b21399 added a 9-year old png encoder from David Eisenberg which I rewrote 
quite a bit to remove all code that handles transparency. With this
highly specialized png writer it is possible to write png images much
faster that with the JRE built-in png writer.
In a second step it can be possible to add concurrency to increase
computation speed further.
 2012-10-23 23:27:41 +02:00
orbiter
276dd6452b removed warnings 2012-10-23 19:08:44 +02:00
Michael Peter Christen
b991685782 Merge branch 'master' of git://gitorious.org/~reger/yacy/bbyacy-rc1 2012-10-23 18:14:58 +02:00
Michael Peter Christen
ea11a1efea fix for highlighting in gsa search 2012-10-23 18:11:49 +02:00
Michael Peter Christen
9eaede50e7 enhanced web structure images 2012-10-23 18:11:19 +02:00
Michael Peter Christen
b7ac1da6a3 gsa results shall have only one title in metadata and that should be the 
visible title in the <title>-tag
 2012-10-23 18:03:12 +02:00
Michael Peter Christen
ae6feb5610 showing the web structure graph as animation in the crawl monitor 2012-10-23 02:50:26 +02:00
reger
87aab9aa7c - fix: with augmented parsing = on; missing metadata in index (like title) due to overwriting metadata by adding multiple result docs from augmentparser with same url 
- fix Document.addsubdocuments: sections might be initialized as Arrays.toList which does not provide the used .addAll methode
   see e.g. http://kamleshkr.wordpress.com/2010/02/17/inside-java-arrays-aslistt-a/
 2012-10-22 22:48:35 +02:00
Michael Peter Christen
39317a6c66 enhanced webstructure image: introduced 
- multiple hosts can be listed (comma-separated) as host argument
- new 'bf'-attribut (branch factor): the maximum number of edges per
node
- the bf-value is computed automatically
- ordering of nodes when the graphic is drawed: mostly the drawing ends
with an limitation eg. number of nodes. When this happens, it should be
ensured that more 'interesting' nodes are painted in advance. This is
now done by sorting all nodes by the number of links they have in de
distant sub-graph.
 2012-10-22 16:23:39 +02:00
sixcooler
47ae7e322e smaller dhtDispatcher.cloudSize 
@Orbiter: we talked about this times ago - please revert if I'm wrong
 2012-10-21 20:05:28 +02:00
sixcooler
57ddd63888 not hold a expensive cache of references for DHT-out,but but load them 
on demand
see: http://forum.yacy-websuche.de/viewtopic.php?f=8&t=4530
 2012-10-21 20:00:36 +02:00
Michael Peter Christen
ea27d2e5f6 fixed more getSolrFieldName usages 2012-10-18 15:21:05 +02:00
Michael Peter Christen
ce0e5b1e17 - more refactoring / private methods 
- fix for usage of custom solr field names
 2012-10-18 15:09:04 +02:00