Marown/yacy_search_server
1
0
Fork 0
mirror of https://github.com/yacy/yacy_search_server.git synced 2024-09-21 00:00:13 +02:00
Code Issues Actions Packages Projects Releases Wiki Activity
10,816 Commits 1 Branch 20 Tags 836 MiB
a1ac4c3b76
Commit Graph

2671 Commits

Author SHA1 Message Date
Michael Peter Christen
7fefebaeca Merge branch 'master' of ssh://git@gitorious.org/yacy/rc1.git 2014-04-17 12:55:38 +02:00
Michael Peter Christen
c2f62e783f - better subgraph handling, less overhead for crawls without the 
webgraph
- usage of crawler crawldepth cache for the linkgraph target depth
computation
 2014-04-17 12:54:18 +02:00
Michael Peter Christen
06afb568e2 new Strategies in Balancer: 
- doublecheck cache now records the crawl depth as well
- doublecheck cache is available from the outside (made static)
- no more need to crawl hosts with lowest depth first, instead all hosts
which have only singleton entries are preferred to reduce the number of
files.
 2014-04-17 12:52:54 +02:00
Michael Peter Christen
1aea01fe5b fix for Table in case that requested file does not exist and paths also 
do not exist
 2014-04-17 12:44:05 +02:00
reger
710054bb37 implement gzip input handling directly in defaultservlet 
(making reference to legacy httpdemon obsolete)
 2014-04-17 03:20:29 +02:00
Michael Peter Christen
9a5ab4e2c1 removed clickdepth_i field and related postprocessing. This information 
is now available in the crawldepth_i field which is identical to
clickdepth_i because of a specific crawler strategy.
 2014-04-16 22:16:20 +02:00
Michael Peter Christen
da86f150ab - added a new Crawler Balancer: HostBalancer and HostQueues: 
This organizes all urls to be loaded in separate queues for each host.
Each host separates the crawl depth into it's own queue. The primary
rule for urls taken from any queue is, that the crawl depth is minimal.
This produces a crawl depth which is identical to the clickdepth.
Furthermorem the crawl is able to create a much better balancing over
all hosts which is fair to all hosts that are in the queue.
This process will create a very large number of files for wide crawls in
the QUEUES folder: for each host a directory, for each crawl depth a
file inside the directory. A crawl with maxdepth = 4 will be able to
create 10.000s of files. To be able to use that many file readers, it
was necessary to implement a new index data structure which opens the
file only if an access is wanted (OnDemandOpenFileIndex). The usage of
such on-demand file reader shall prevent that the number of file
pointers is over the system limit, which is usually about 10.000 open
files. Some parts of YaCy had to be adopted to handle the crawl depth
number correctly. The logging and the IndexCreateQueues servlet had to
be adopted to show the crawl queues differently, because the host name
is attached to the port on the host to differentiate between http,
https, and ftp services.
 2014-04-16 21:34:28 +02:00
Michael Peter Christen
075b6f9278 refactoring of the crawl balancer: the balancer is turned into an 
interface and the old balancer class is moved into LegacyBalancer to
make room for a fresh implementation of a crawl balancer.
 2014-04-14 13:32:35 +02:00
Michael Peter Christen
8470dfe3f8 Merge branch 'master' of ssh://git@gitorious.org/yacy/rc1.git 2014-04-14 12:17:52 +02:00
reger
46016fa153 autoupdate fails to download latest release (1.71) due to default release blacklist 
- removed the default version blacklist regex from init (for future versions)

!!!  left existing update  blacklist setting untouched !!! 
(existing installation wanting autoupdate for 1.71 need to change blacklist in ConfigUpdate_p.html)

- moved old blacklist patch to migration.java
 2014-04-13 07:32:32 +02:00
Michael Peter Christen
8aeef73d49 fix for virtual root nodes 2014-04-11 15:12:34 +02:00
Michael Peter Christen
7c7fbb9818 find depth-matches also for edge targets 2014-04-11 12:27:21 +02:00
Michael Peter Christen
dd12dd392f introduction of a data structure for HyperlinkEdges which should use 
less memory as it does no double-storage of source links for each edge
of the graph.
 2014-04-11 12:09:33 +02:00
Michael Peter Christen
6ea8bb7348 using MultiProtocolURL for edge data which is faster (hash computation 
is now much easier) and smaller in size
 2014-04-11 10:58:37 +02:00
Michael Peter Christen
b21c208b4d enhanced hashcode computation for MultiProtocolURL 2014-04-11 10:23:48 +02:00
Michael Peter Christen
ce1d1b2fa0 fix for maximum tag length in parser 2014-04-11 09:56:44 +02:00
Michael Peter Christen
17e0956312 refactoring of SystemLoad calls (only one backend tool) 2014-04-11 09:25:18 +02:00
Michael Peter Christen
a37d067692 refactoring 2014-04-10 23:46:35 +02:00
orbiter
95780eed32 Merge branch 'master' of git@gitorious.org:yacy/rc1.git 2014-04-10 21:40:54 +02:00
Michael Peter Christen
67beef657f strong redesign of html parser: object recursion is now made using a 
stack on html tag objects, not using a recursive parse-again method
which may cause bad performance and huge memory allocation. The new
method also produced better parsed image objects with exact anchor text
references.
 2014-04-10 18:58:03 +02:00
Michael Peter Christen
6bd8c6f195 fix for wrong status codes of error pages 2014-04-10 09:08:59 +02:00
Michael Peter Christen
9e503b3376 also delete the robots.txt file from the cache when a new crawl is 
started
 2014-04-09 21:59:54 +02:00
orbiter
67501c9dda Merge branch 'master' of git@gitorious.org:yacy/rc1.git 2014-04-09 19:58:54 +02:00
Michael Peter Christen
1c21b3256d fix for robots.txt handling: delete old entry before starting a new 
crawl.
 2014-04-09 18:33:48 +02:00
orbiter
c250fac9f4 linkstructure refactoring to get more options for clickdepth analysis 2014-04-09 17:52:51 +02:00
Michael Peter Christen
8068e68474 Merge branch 'master' of ssh://git@gitorious.org/yacy/rc1.git 2014-04-09 12:45:15 +02:00
Michael Peter Christen
bd886054cb new structure and enhancements for link graph computation: 
- added order option to solr queries to be able to retrieve document
lists in specific order, here: link length
- added HyperlinkEdge class which manages the link structure
- integrated the HyperlinkEdge class into clickdepth computation
- extended the linkstructure.json servlet to show also the clickdepth
and other statistic information
 2014-04-09 12:45:04 +02:00
reger
f326a67561 fix: typo in default charset in metadata2solr 
update pom and NB build to Solr 4.7.1 libs
 2014-04-06 22:31:22 +02:00
Michael Peter Christen
df138084c0 do solr optimization independently from memory and load constraints: 
- not doing an optimization will likely cause a too many files exception
- without optimization performance will be even worse which would
prevent optimization in the future as well (prevent a deadlock
situation)
 2014-04-06 11:04:23 +02:00
Michael Peter Christen
ebd44a7080 replaced solr 4.6.1 with solr 4.7.1 and added index migration to 
lucene_47
 2014-04-06 10:45:03 +02:00
Michael Peter Christen
734778c0c8 fixed a time-out problem in the default servlet which is also a logging 
problem because the error log showed the wrong reason (file not found)
instead the actual reason (time-out).
 2014-04-04 15:27:29 +02:00
Michael Peter Christen
466d90ad42 fixed a problem with resource observer; probably coming from uncatched 
exceptions within the apache library which appear only in concurrency
environments.
 2014-04-04 15:26:39 +02:00
Michael Peter Christen
e8ddd415a8 enhanced the new link structure graph 2014-04-04 14:43:54 +02:00
Michael Peter Christen
926d28dd3f fixed a bug which prevented crawl starts after a network switch 2014-04-04 14:43:35 +02:00
Michael Peter Christen
3ce8eff21b another fix for inbound/outbound detection 2014-04-04 12:41:59 +02:00
Michael Peter Christen
d4b5c457e4 NPE fix 2014-04-04 12:34:34 +02:00
Michael Peter Christen
36a66b0704 fix for parsing of numeric value in case that boolean values are given 2014-04-04 11:59:51 +02:00
orbiter
41730c8048 better logging in template engine: shows filename of servlets where 
errors in templates occur
 2014-04-04 10:55:46 +02:00
orbiter
3c1274057d fixed thread dump in case of wrong seeds 2014-04-04 10:54:56 +02:00
orbiter
18f9c40302 moved Edge class out of linkstructure servlet as this does not work on 
non-eclipse driven environments (all non-dev cases)
 2014-04-04 10:54:11 +02:00
orbiter
de95e5e524 reduced search activity corona strength in network image 2014-04-04 10:08:44 +02:00
reger
da413af664 move baseurl after parsing orig source in urlproxyservlet 
to calculate absolute href links for rewrite from unmodified source.
 2014-04-04 03:11:16 +02:00
reger
af6ad20728 fix: remove obsolete ref to yacy.home 
(use Switchboard instead)
 2014-04-04 02:45:04 +02:00
Michael Peter Christen
74ab094587 fix for solr query size; too many documents had been retrieved in case 
that less than _pagesize_ had been requested.
 2014-04-03 13:42:10 +02:00
Michael Peter Christen
c64c10ef00 Merge branch 'master' of ssh://git@gitorious.org/yacy/rc1.git 2014-04-03 01:58:06 +02:00
Michael Peter Christen
48fbfa60c1 bugfix to inbound/outbound identification 2014-04-03 01:21:43 +02:00
reger
227c42bc96 eleminate obsolete URIMetaDataRow class 
by joining it with/into URIMetaDataNode.
 2014-04-03 00:35:15 +02:00
Michael Peter Christen
cca851a417 introduced new solr field crawldepth_i which records the crawl depth of 
a document. This is the upper limit for the clickdepth_i value which may
be shorter in case that the crawler did not take the shortest path to
the document.
 2014-04-02 23:37:01 +02:00
orbiter
b1ba764d81 fix for first start options and added german translation for popup texts 2014-04-02 17:10:59 +02:00
orbiter
429a874222 - added COLS field in GSA response (non-gsa standard by customer 
request)
- updated document link in GSA response writer
 2014-04-02 16:05:44 +02:00
Michael Peter Christen
1b9ec9a1c5 - added popover to p2p/stealth mode button to explain the peer mode and 
privacy issues.
- added popover to first-time use case to explain that specific servlets
are only visible after customization and/or crawl starts
 2014-04-02 13:33:43 +02:00
Michael Peter Christen
62a36fa584 Merge branch 'master' of ssh://git@gitorious.org/yacy/rc1.git 2014-04-02 03:27:08 +02:00
reger
c9f92abddc fix: application link count 
(URIMetadataNode)
 2014-04-02 03:21:51 +02:00
Michael Peter Christen
a267c46e1a Merge branch 'master' of ssh://git@gitorious.org/yacy/rc1.git 2014-04-02 02:35:58 +02:00
Michael Peter Christen
5b83887da8 npe fix 2014-04-02 02:34:55 +02:00
Michael Peter Christen
63c9fcf3e0 free configuration of postprocessing clickdepth maximum depth and time 2014-04-02 02:34:39 +02:00
Michael Peter Christen
39b641d6cd added tutorial mode - some menu items will only appear if you 'qualify' 
for them. Thus, the first-time user will only see four menu items. The
other items will unfold as the user interacts.
 2014-04-02 02:33:17 +02:00
sixcooler
f06775850f fix receiving DHT / parse pultipart 
+ another close to fix possible resource leak warning
 2014-04-02 01:24:15 +02:00
reger
49e76a1c55 make use of detected charset in htmlParser if none is given. 2014-04-01 04:02:34 +02:00
reger
e11504309f adding a hint to javascript browser short cut on Url-Proxy page (AugmentedBrowsing_p.html) 2014-03-30 05:11:42 +02:00
reger
b12200cafe alternative UrlProxyServlet (for /proxy.html) using different url rewrite rules 
- use JSoup parser for selective rewrite of html body <a href=  links only,
instead of regex which rewrites also header href/src links
- this improves display of pages which use header <base> tag
- tags with src attribute are taken from original location (like css) improving display and are not routed trough the indexer
Disadvantage: scripting links will drop out of proxy

Setting of the servlet through web.xml exclusivly (in case one would like to quickly switch back to the YaCyProxyServlet,
leaving the existing code of YaCyProxyServlet untouched available)
 2014-03-30 04:04:02 +02:00
reger
2953ebe701 fix: port in local target adress 
& button style
 2014-03-29 00:34:01 +01:00
Michael Peter Christen
fda591695c fixed visibility of custom icon 2014-03-28 17:25:39 +01:00
Michael Peter Christen
a9b9950d7f Merge branch 'master' of ssh://git@gitorious.org/yacy/rc1.git 2014-03-28 14:48:32 +01:00
Michael Peter Christen
b488f33975 added close to fix possible resource leak warning 2014-03-28 14:34:49 +01:00
Michael Peter Christen
56710ecb26 prevent opening of new files as that could be a cause for the latest 
too-many-open-files exception. The old file is just truncated if the
table is cleaned.
 2014-03-28 14:31:43 +01:00
Michael Peter Christen
8b44fcf0f4 added missing @Override annotation 2014-03-28 13:48:37 +01:00
reger
d7055904a6 fix: proxyservlet path header setting 2014-03-28 02:05:58 +01:00
Michael Peter Christen
e515dd460d added linkscount_i and linksnofollowcount_i to the default solr schema 2014-03-27 23:36:08 +01:00
Michael Peter Christen
1a764135be one more Thread Dump fix for new bootstrap css style 2014-03-27 23:01:28 +01:00
Michael Peter Christen
bb21d825f9 fix for thread dump line spacing 2014-03-27 22:13:37 +01:00
Michael Peter Christen
cbdfef7ce1 changed protocol facet to show also all other counts if one facet is 
selected
 2014-03-27 13:29:14 +01:00
reger
b9056ef2db remove unused private header entries (HeaderFramework) 
X_YACY_ORIGINAL_REQUEST_LINE
X_YACY_KEEP_ALIVE_REQUEST_COUNT
CONNECTION_PROP_REQUESTLINE
 2014-03-26 23:28:19 +01:00
sixcooler
6d16fa993d make transparent proxy handle https-connections: 
the implemented handle for connect did not work for me - so lets try the
connectHandler
 2014-03-26 20:01:15 +01:00
Michael Peter Christen
61ad194065 fix for source and target clickdepth in webgraph index 2014-03-26 16:00:05 +01:00
Marc Nause
809b4e1fd9 Team added support for URLs with unicode characters in host part to 
blacklist. Punycode is used to handle unicode characters.
 2014-03-25 22:14:54 +01:00
reger
b126b9ba17 add some InputFileStream close at end of reads 
to make sure file is released
 2014-03-24 02:32:17 +01:00
reger
ca7444dbdf limit filetype nav to known extension also on image/media search 
- on text search we limit filetype nav already to known extension, apply filter to image search
 2014-03-23 23:10:29 +01:00
reger
651d057e93 surrogate import translate dc:language 3-char codes 
OAI records often use 3-char language codes, start converting some 3-char lang's to the internal ISO639-1 2-char code
 2014-03-23 00:40:36 +01:00
orbiter
22618e3ba2 Merge branch 'master' of ssh://git@gitorious.org/yacy/rc1.git 2014-03-21 20:28:50 +01:00
orbiter
01989f6af9 restrict write buffer size to a limit 2014-03-21 20:28:34 +01:00
Michael Peter Christen
d1091e79f8 - added stealth button to navigation menu 
- more fixes to progress bar
 2014-03-21 18:01:26 +01:00
reger
c297de5145 remove check for unused virtual path /currentyacypeer/ 
-  del jqueryheader.template (not used)
 2014-03-21 03:02:19 +01:00
orbiter
3c8d6e1eee added adminAccount switch to ConfigAccounts_p servlet to switch on 
protection of all pages; some refactoring as well
 2014-03-20 22:11:49 +01:00
orbiter
7d24bcb98d added flag to require that all web pages, even such without a "_p" 
extension require authorization. (default off)
 2014-03-20 19:09:47 +01:00
Michael Peter Christen
7a6658abec removed synchronization in embedded solr connection (that was probably 
a mistake?)
 2014-03-19 16:21:03 +01:00
Michael Peter Christen
a7d4379ef9 fixed shutdown of solr cores in case that more than one local core is to 
be closed (this happens if webgraph is enabled and the index is dumped
using /IndexControlURLs_p.html
 2014-03-19 12:23:40 +01:00
Michael Peter Christen
453bfd0f17 removed unused variables and warnings 2014-03-19 09:29:01 +01:00
Michael Peter Christen
05655d98df Merge branch 'master' of ssh://git@gitorious.org/yacy/rc1.git 2014-03-17 11:57:01 +01:00
reger
9f02d2c47b fix: remove link to triplestore in Vocabulary_p (triplestore does not longer exist) 
- should be investigated in more detail to look for additional implications
Remove "yacyaction" from proxyservlet as it was only needed for removed interaction routines.
 2014-03-16 22:11:19 +01:00
reger
81a846ec33 fix: set YaCy CONNECTION_PROP_HOST Header in ProxyServlet to host incl. port 2014-03-16 20:51:32 +01:00
reger
251be9ecfa remove unused ProxySettings ref. from loader 
clean unused whois test code
 2014-03-16 05:19:01 +01:00
reger
82dc815af9 cleanup: remove unrelated and unused code 2014-03-16 00:15:12 +01:00
Michael Peter Christen
85a427ec54 support for multiple sitemaps in robots.txt 2014-03-14 13:33:23 +01:00
reger
a373fb717d remove more unused from legacy server.http 
- triggerOnlineAction not used
- useTemplateCache not used
 2014-03-14 03:12:04 +01:00
reger
749d020aeb remove redundant url string manipulation in HTTPDProxyHandler 
(still used by ProxyServlet)
 2014-03-14 02:24:12 +01:00
reger
612294cf84 use servletPath in ProxyServlet instead of fixed name 
to allow servlet-mapping via web.xml
 2014-03-13 02:46:05 +01:00
reger
1d01672bd3 fix DCEntry.getIdentifier 
on successful url parameter
 2014-03-12 23:35:57 +01:00
Michael Peter Christen
b08375da33 fix for bad/missing values of size_i 2014-03-11 09:51:04 +01:00
reger
6306d28a6a OAI import get multivalued keywords (dc:subject) 2014-03-09 03:15:35 +01:00