Marown/yacy_search_server
1
0
Fork 0
mirror of https://github.com/yacy/yacy_search_server.git synced 2024-09-21 00:00:13 +02:00
Code Issues Actions Packages Projects Releases Wiki Activity
9,047 Commits 1 Branch 20 Tags 836 MiB
7bec253bb0
Commit Graph

9047 Commits

This Branch
This Branch
All Branches
Author SHA1 Message Date
Michael Peter Christen
7bec253bb0 Merge branch 'master' of ssh://git@gitorious.org/yacy/rc1.git 2012-11-04 09:21:58 +01:00
Michael Peter Christen
d88eb657fd Merge branch 'master' of git://gitorious.org/~reger/yacy/bbyacy-rc1 2012-11-04 09:21:21 +01:00
orbiter
354ef8000d - added 'deleteold' option to crawler which causes that documents are 
deleted which are selected by a crawl filter (host or subpath)
- site crawl used this option be default now
- made option to deleteDomain() concurrency
 2012-11-04 02:58:26 +01:00
Michael Peter Christen
19d1f474ce host browser now shows also number of pending files per subdirectory + 
bugfixes
 2012-11-02 14:40:02 +01:00
Michael Peter Christen
75dd706e1b update to HostBrowser: 
- time-out after 3 seconds to speed up display (may be incomplete)
- showing also all links from the balancer queue in the host list (after
the '/') and in the result browser view with tag 'loading'
 2012-11-02 13:57:43 +01:00
Michael Peter Christen
e2c4c3c7d3 migration to solr 4.0.0 2012-11-02 12:29:48 +01:00
Michael Peter Christen
b764de424a code cleanup 2012-11-02 10:28:32 +01:00
Michael Peter Christen
69aa39d664 update to libraries required by solr 4.0.0 2012-11-02 10:27:44 +01:00
Michael Peter Christen
9330ad4838 - fixed the delete option in host browser 
- added a delete method which can be used to delete a full subpath in
solr.
 2012-11-02 01:22:31 +01:00
Michael Peter Christen
a63179f3f9 added the MIME attribute for the R tag in GSA search result writer 2012-11-02 00:14:29 +01:00
Michael Peter Christen
40df2fd193 added the host browser as link to search results. that means you can 
select a browsing position after a search is done on the search results.
 2012-11-01 21:38:05 +01:00
Michael Peter Christen
1168d09de8 more refactoring - integrated the code of SnippetProcess into 
SearchEvent
 2012-11-01 17:40:06 +01:00
Michael Peter Christen
6629e37685 tried to clean up the search process mess 2012-11-01 17:16:43 +01:00
Michael Peter Christen
c5f67a5d6d fixed a problem with local search from solr results: now all results 
from solr are shown (again)
 2012-11-01 10:22:22 +01:00
sixcooler
02957d5982 missing license-files 
(sorry I didn't commit theses files by mistake)
 2012-10-31 23:47:08 +01:00
Michael Peter Christen
16216c2344 added missing libraries 2012-10-31 23:29:47 +01:00
sixcooler
9d062873d2 bump to httpclient-4.2.2 2012-10-31 19:09:48 +01:00
Michael Peter Christen
f8f05ecba7 - added a delete button in host browser to delete a complete subpath 
- removed storage of default collection name - default is now "user"
- made stacking of crawl start points concurrently
 2012-10-31 17:44:45 +01:00
Michael Peter Christen
0716a24737 added more / all new crawl profile fields into crawl profile editor 2012-10-31 15:13:05 +01:00
Michael Peter Christen
4a14122ba7 in case that a crawl profile has a collection assigned, use the 
collection to show a name in the web interface. This should prevent that
much too long names make the interface unusable.
 2012-10-31 14:08:33 +01:00
Michael Peter Christen
0fe8be7981 enhaced data structures for balancer and latency computation which 
should produce a bit better prognosis about forced waiting times.
 2012-10-30 17:30:24 +01:00
Michael Peter Christen
ac9540dfb6 removed options for stopwords which are not used 2012-10-30 12:36:36 +01:00
Michael Peter Christen
ce3fed8882 added the Google Search Appliance (GSA) api interface to the main menu. 
See:
https://developers.google.com/search-appliance/documentation/68/xml_reference#request_overview
 2012-10-30 12:27:22 +01:00
Michael Peter Christen
b2ffd49817 less latency 2012-10-30 12:26:32 +01:00
Michael Peter Christen
0833937c1c better balancing and duetime-cumputation also for no-delay intranet 
hosts
 2012-10-30 11:28:49 +01:00
Michael Peter Christen
c326aa8f67 disabled writing new entries to crawl stacks to prevent that a domain 
with many documents block refreshing of the crawl queue
 2012-10-29 22:26:52 +01:00
Michael Peter Christen
6905182d41 - fix for number of words log message 
- adding meta:refresh also to crawler stack
 2012-10-29 21:42:31 +01:00
Michael Peter Christen
c25d7bcb80 - added concurrency for robots.txt loading 
- changed data model for domain counter
 2012-10-29 21:08:45 +01:00
Michael Peter Christen
a94c537afc fixed getSize() which can use the cache size while the crawl is running 2012-10-29 11:56:07 +01:00
Michael Peter Christen
96912c9471 enhancement to solr caching: consider that during a get() the document 
is not in solr but the cache points out that a commit is needed to get
the document.
 2012-10-29 11:35:24 +01:00
Michael Peter Christen
a87811bc38 more auto-commit calls when a search interface is opened, but not when a 
search is done there to prevent blocking during search-time.
 2012-10-29 11:27:13 +01:00
Michael Peter Christen
3d3d654e88 if a network configuration is choosed which does not allow DHT and no 
P2P communication is in robinson mode) then some menu entries are
disabled which have no use in this mode.
 2012-10-29 01:51:19 +01:00
Michael Peter Christen
2d9e577ad0 replaced the custom robots.txt loader by the standard http loader 2012-10-28 22:48:11 +01:00
Michael Peter Christen
799d71bc67 enhanced solr caching: 
- increased cache size which is needed for longer solr commit time
- speed hacks on cache write code
 2012-10-28 20:31:29 +01:00
Michael Peter Christen
a33e2742cb - removed unnecessary synchronized and deadlock in crawler 
- removed problem with monitoring object on Balancer.wait
- added missing user agent settings
 2012-10-28 19:56:02 +01:00
orbiter
8952153ecf update to Balancer algorithm: 
- create a load list from the current list of known hosts
- do not create this list for each Balancer.pop access
- create the list from those hosts which have a zero-waiting time
- select 1/3 from that list which have the most urls waiting
- get hosts from the wainting list in random order
- fixes for some delta-time computations
- always load all urls from hosts which have never been loaded before
 2012-10-28 13:24:49 +01:00
orbiter
354f0d9acd moved static method from ClusteredScoreMap to MapDataMining because it 
was not used in the ClusteredScoreMap class but only in MapDataMining
 2012-10-28 11:29:53 +01:00
reger
722a447b0d - optimize code of augmented parsing to enhence document tags 
- commented out augmentedparser.analyse (not function implemented yet)
- adjust init of document title list to always use same list type
 2012-10-26 18:50:45 +02:00
Michael Peter Christen
8e1248ffe3 force a commit in advance of a search for the administrator to get most 
recent results even if commit time is high and an indexing is ongoing.
 2012-10-26 15:35:42 +02:00
Michael Peter Christen
3b48c78190 added an option to force a commit to solr. 
may be used by a search front-end in case that the commitWithinMs time
is too short to get recently indexed documents.
 2012-10-26 07:39:07 +02:00
sixcooler
2d972f289a rise commitWithinMs to default-value from SwitchBoard 
(result in lower hd-io)

no dots in memory-graph (there are to much of them)
 2012-10-26 02:12:45 +02:00
orbiter
8fde1dd3b6 another performance and memory hack to graphics: this makes it possible 
to produce a 100-Megapixel png network graphic image on my 6 year old
laptop in standard configuration in 10 seconds.
 2012-10-25 21:40:27 +02:00
Michael Peter Christen
1baf498d59 - show more lines in online log 
- reverse order is default now
 2012-10-25 18:38:39 +02:00
Michael Peter Christen
55bdafbaf1 more image processing hacks 2012-10-25 18:20:05 +02:00
Michael Peter Christen
f2d0418218 because the new PngEncoder had a problem with the PixelGrabber which is 
caused by a JRE bug, the PixelGrabber had to be circumvented using an
own frame buffer which can be read without a PixelGrabber. This resulted
in ultra-fast and much less memory-consuming transformation. YaCy images
are now generated really fast!
 2012-10-25 17:59:20 +02:00
Michael Peter Christen
d5d64019e5 - added a method for the RasterPlotter to draw arrow endings to lines 
- replaced the dot in the NetworkGraph with arrows
- enhanced the image drawing speed using pre-computed color values
- added more attention for OOM cases during very large image painting
 2012-10-25 16:05:04 +02:00
Michael Peter Christen
342543a6c4 fix for host browser 2012-10-25 10:23:43 +02:00
Michael Peter Christen
85ca07b90e when a new crawl is started, an equal crawl, if still running, is 
terminated and the corresponding crawl profile is deleted (this also
clears the crawl queue entries for that crawl profile)
 2012-10-25 10:20:55 +02:00
Michael Peter Christen
906e51214a the web structure image shows the pivot dot in a different color 2012-10-25 10:18:28 +02:00
Michael Peter Christen
b3ffcde0c7 - prepared PngEncoder for concurrency: PixelGrabber.grabPixels is the 
main time-consuming process. This shall be done in concurrency.
- added concurrent processes to call the PixelGrabber and framework to
do that (queues)

It is now possible to create 4k-Images (3840x2160) i.e. with the Network
Graphics servlet
 2012-10-24 02:08:51 +02:00