Marown/yacy_search_server
1
0
Fork 0
mirror of https://github.com/yacy/yacy_search_server.git synced 2024-09-19 00:01:41 +02:00
Code Issues Actions Packages Projects Releases Wiki Activity
11,002 Commits 1 Branch 20 Tags 836 MiB
29d1945c16
Commit Graph

7230 Commits

Author SHA1 Message Date
Michael Peter Christen
53948da7d0 tried to make last_modified recognition smarter 2014-05-22 00:28:51 +02:00
Michael Peter Christen
2d03037965 'Last-Modified', not 'Last-modified' according to 
http://www.w3.org/Protocols/rfc2616/rfc2616-sec14.html
 2014-05-21 23:21:31 +02:00
Michael Peter Christen
3dc5fb0050 fix for operator precedence bug (cast binds stronger than bitwise AND) 
in peer hash hashing. This should not change anything if java casts long
to int by masking with 0xFFFFFFFFL but you never know. The important
thing is, that the hashCode() should not return numbers that have the
same order as the hash code order because hashing of seeds is used to
remove the order in some places.
 2014-05-21 18:37:52 +02:00
Michael Peter Christen
6634b5b737 debug code for index distribution testing 2014-05-21 18:20:16 +02:00
orbiter
49e344e8d9 Merge branch 'master' of git@gitorious.org:yacy/rc1.git 2014-05-21 09:28:55 +02:00
orbiter
7705e36703 fix for latest generic warning fix 2014-05-21 09:28:23 +02:00
sixcooler
10326892a8 avoid erros from ConnectHandler, correction for #6d16fa9 2014-05-21 03:04:07 +02:00
orbiter
97983ba89f fixed generics warnings for generic array instantiation that appeared 
after migration to Java 7
 2014-05-20 21:50:16 +02:00
sixcooler
830057d788 lower Segment-size (hope to get Segments of 10GB) 
see:
http://forum.yacy-websuche.de/viewtopic.php?f=5&t=5216&p=30036#p30034
 2014-05-19 17:55:03 +02:00
orbiter
c028ae9b09 Merge branch 'master' of git@gitorious.org:yacy/rc1.git 2014-05-18 21:21:17 +02:00
reger
e31493e139 "Use remote proxy for yacy" has no function, remove option and related config item 
see/fix bug http://mantis.tokeek.de/view.php?id=23
http://mantis.tokeek.de/view.php?id=189
 2014-05-17 23:36:59 +02:00
orbiter
181784a5cb Merge branch 'master' of git@gitorious.org:yacy/rc1.git 2014-05-15 08:06:59 +02:00
reger
0587077d06 cleanup obsolete and not used serverswitch Authentify code 
as auth is mostly delegated to Jetty container.
 2014-05-14 23:13:49 +02:00
orbiter
c9f66be20b move unnecessary nested else out of condition 2014-05-13 22:31:12 +02:00
orbiter
0d8072aa99 removed warnings 2014-05-13 22:29:05 +02:00
orbiter
88f4af90da removed warnings 2014-05-13 22:27:31 +02:00
orbiter
0f425e01ca another circle computation enhancement 2014-05-13 21:30:47 +02:00
reger
a8d162810c Exclude = from percent-encoding in MultiProtocolURL 
fix http://mantis.tokeek.de/view.php?id=185 and http://mantis.tokeek.de/view.php?id=280
 2014-05-13 02:33:35 +02:00
reger
024f8e9b33 fix truncated urls containing "," 
adressing http://mantis.tokeek.de/view.php?id=58

Exclude comma from percent-encoding in MultiProtocolURL (see  RFC 1738 2.2  and  RFC 3986 2.2)
 2014-05-13 01:50:15 +02:00
Michael Peter Christen
9112f0a2df enhanced circle tool initialization 2014-05-12 16:21:24 +02:00
Michael Peter Christen
a1ac4c3b76 automatically clear graphics cache 2014-05-12 15:45:25 +02:00
Michael Peter Christen
505f58c79c enhanced circle computation time and memory footprint 2014-05-12 15:34:56 +02:00
reger
cd8c0dbda9 assign serialVersionUID for proxyservlet, too. 2014-05-11 03:51:47 +02:00
reger
b300d7f4ce set serialVersionUID on urlproxyservlet to skip compiler warning 
- remove commented out code
 2014-05-11 03:31:07 +02:00
reger
e9060d31bd update to Jetty 9 
besides adjustments in code it makes the servlet settings in web.xml significant.
This applies to solr, gsa and proxy servlet. There is no longer a default setup in code during init (as jetty 9 checks for double definition).
 2014-05-11 01:53:11 +02:00
reger
1432a817dd respect "index media" switched off in CrawlStartExpert.html 
fix http://mantis.tokeek.de/view.php?id=64
 2014-05-08 22:21:24 +02:00
orbiter
39e1913585 next development step: migration to java 1.7 
This includes also a small code change to test generic type inference, a
java 1.7 feature
 2014-05-08 07:41:11 +02:00
Michael Peter Christen
4e734815e8 enhanced snippets: remove lines which are identical to the title and 
choose longer versions if possible. Prefer the description part.
 2014-05-06 16:48:50 +02:00
Michael Peter Christen
e84e07399a Merge branch 'master' of ssh://git@gitorious.org/yacy/rc1.git 2014-05-06 14:51:57 +02:00
orbiter
89f76da24b Merge branch 'master' of git@gitorious.org:yacy/rc1.git 2014-05-06 05:38:38 +02:00
sixcooler
390f03e041 o not check for segments-count on optimize: 
this is also done in Solr and our getSegmentsCount() does not return
up-to-date values
 2014-05-05 13:24:41 +02:00
reger
8a7c68e4c7 content of surrogates/out never accessed (remove) 
After import the conent is never accessed but may take up a lot of disk space,
also the getLoadedOAIServer (which lists the files in surrogate out) is not used.
Making the surrogate.out obsolete. Removed keeping of xmls after import.
 2014-05-04 09:29:07 +02:00
sixcooler
b8cee9b7d8 remove tables from tabletracker on close to avoid lots of dead entrys in 
/PerformanceMemory_p.html
 2014-05-02 22:55:47 +02:00
reger
1600414450 fix NPE on continuing crawls after YaCy restart 
(Agent is then nulll)
 2014-05-02 19:32:09 +02:00
Michael Peter Christen
229f2248b8 added configuration option for maxmimum load and minimum ram for 
postprocessing
 2014-04-30 13:26:32 +02:00
orbiter
f15c832587 Merge branch 'master' of git@gitorious.org:yacy/rc1.git 2014-04-30 07:42:52 +02:00
Marc Nause
c97da1a0d8 First draft of a blacklist API. 2014-04-30 00:48:38 +02:00
Michael Peter Christen
d4f65833a1 Merge branch 'master' of ssh://git@gitorious.org/yacy/rc1.git 2014-04-29 19:51:01 +02:00
Michael Peter Christen
c1c1be8f02 fix for slow crawling and better logging in balancer 2014-04-29 19:50:33 +02:00
Michael Peter Christen
3acf416335 npe fix 2014-04-29 19:24:05 +02:00
reger
2eb7682772 add html5 audio/video <source> tag to html content scraper 
- <source src=.. type=..> tag content is added to embed collection
 2014-04-29 00:41:29 +02:00
reger
0b6db04e40 fix contentscraper img height/width parsing 
prevent numberformat exception on common "100px" property

- include in test case
 2014-04-28 04:59:47 +02:00
reger
ffc5b75c73 optimize and fix lat / lon assignment 2014-04-27 20:52:06 +02:00
reger
9313447de2 reimplement tighter lat/lon calc in URIMetadataNode 
from old MetadataRow, considering http://mantis.tokeek.de/view.php?id=272
 2014-04-27 18:20:33 +02:00
reger
d812f80784 add exit proxy link to UrlProxy 
on proxied pages a link to exit proxy is added to top of page.
Link text can be configured in web.xml init-parameter (see default/web.xml). If missing no link is displayed.
 2014-04-26 22:27:59 +02:00
reger
78d08998db throw MalformedURLException on unknown protocol 
on other than the supported   http https ftp file smb \\  mailto
 2014-04-26 01:30:51 +02:00
reger
bb8181b2be fix: resolve url without path but searchpart 
e.g. http://yacy.net?q=test was resolved as host "yacy.net?q=test" now host="yacy.net" path="/"
fixes http://mantis.tokeek.de/view.php?id=47

added test case for getHost
 2014-04-25 20:15:55 +02:00
orbiter
a3542f29b4 npe fix 2014-04-25 09:26:20 +02:00
orbiter
c48d2a2a02 npe fix 2014-04-25 09:23:10 +02:00
reger
121d25be38 recover sax fatal error on OAI-PMH import of xml with entity error 
this allows to continue loading next resumptionToken even if import file caused sax parser error
fix http://mantis.tokeek.de/view.php?id=63
 2014-04-25 01:05:28 +02:00
reger
81dc2aa536 add current css to HTMLResponseWriter to fix metadata view 
(using css from metas.template except js links)
 2014-04-23 23:41:10 +02:00
orbiter
2fd8a0ead6 Merge branch 'master' of git@gitorious.org:yacy/rc1.git 2014-04-23 23:13:23 +02:00
orbiter
8e5ce7cd51 fixed a situation where finished crawls had not been detected. 2014-04-23 23:13:07 +02:00
orbiter
2f63bd0261 enhanced Host Balancer strategy: fair round robin 2014-04-23 23:11:37 +02:00
orbiter
0c88a32c36 do not apply lazy value instantiation for numeric or boolean values 
because that is misleading and confusing in case of 0- or false-values
and may cause NPEs in retrieval functions.
 2014-04-23 08:41:36 +02:00
orbiter
8e04030596 in case of short memory, do not cut down robinson peers to 1, just 
reduce by 50%
 2014-04-23 08:37:19 +02:00
reger
86f6975edc exclude html tags in in/outboundlinks_anchortext_txt parsed text 
- some outboundlinks_anchortext_txt in index contain e.g. <span>text</span> or more tags,
remove all tags for text property (inline img tags are still parsed)
- added test case for above (to htmlParserTest)
- fix solr test case
 2014-04-23 00:55:16 +02:00
orbiter
ccb1864d55 catch IllegalArgumentException for wrong process types (that is needed 
for migrations when new process types are introduced or disappear)
 2014-04-22 23:14:05 +02:00
orbiter
4ee4ba1576 fix for NPE in IndexCreateParserErrors_p.html caused by bad handling of 
lazy value instantiation of 0-value in crawldepth_i
 2014-04-22 19:48:49 +02:00
orbiter
12ba890205 removed warnings 2014-04-22 19:35:15 +02:00
reger
d51f9cc863 add custom Jetty errorhandler 
to provide custom error page footer line
- remove redundant mime check in UrlProxyServlet
 2014-04-21 17:28:21 +02:00
reger
c193a02023 defer creation of new ArrayList after possible early return 
(to skip not used object allocation)
 2014-04-21 17:16:06 +02:00
reger
727dfb5875 refactore URIMetadataNode to further unify interaction with index 
-  URIMetadataNode extending SolrDocument
- use language as stored (String), reducing conversion to string
- optimize debug code in transferIndex
 2014-04-20 01:41:30 +02:00
reger
79e7947442 - remove empty http0_9 status text array 
and unused default_charset = ISO-8859-1
 2014-04-18 22:03:16 +02:00
reger
2dabe2009d - remove unused manual http KeepAlive config 
(reducing references to obsolete httpdemon)
- add port info to settings_http
 2014-04-18 19:57:35 +02:00
Michael Peter Christen
5746aae3db add canonical links to the same crawldepth, not the next crawldepth 2014-04-18 06:51:46 +02:00
Michael Peter Christen
74ab5ef9fa increased runtime for postprocessing query job 2014-04-18 06:51:10 +02:00
Michael Peter Christen
8b32dd5f9e special strategy for balancer: do not remove targets with zero wait time 
from the queue
 2014-04-18 06:50:07 +02:00
Michael Peter Christen
9c6228d948 fix for deadlocks in crawler 2014-04-17 16:58:17 +02:00
Michael Peter Christen
10cf8215bd added crawl depth for failed documents 2014-04-17 13:21:43 +02:00
Michael Peter Christen
7fefebaeca Merge branch 'master' of ssh://git@gitorious.org/yacy/rc1.git 2014-04-17 12:55:38 +02:00
Michael Peter Christen
c2f62e783f - better subgraph handling, less overhead for crawls without the 
webgraph
- usage of crawler crawldepth cache for the linkgraph target depth
computation
 2014-04-17 12:54:18 +02:00
Michael Peter Christen
06afb568e2 new Strategies in Balancer: 
- doublecheck cache now records the crawl depth as well
- doublecheck cache is available from the outside (made static)
- no more need to crawl hosts with lowest depth first, instead all hosts
which have only singleton entries are preferred to reduce the number of
files.
 2014-04-17 12:52:54 +02:00
Michael Peter Christen
1aea01fe5b fix for Table in case that requested file does not exist and paths also 
do not exist
 2014-04-17 12:44:05 +02:00
reger
710054bb37 implement gzip input handling directly in defaultservlet 
(making reference to legacy httpdemon obsolete)
 2014-04-17 03:20:29 +02:00
Michael Peter Christen
9a5ab4e2c1 removed clickdepth_i field and related postprocessing. This information 
is now available in the crawldepth_i field which is identical to
clickdepth_i because of a specific crawler strategy.
 2014-04-16 22:16:20 +02:00
Michael Peter Christen
da86f150ab - added a new Crawler Balancer: HostBalancer and HostQueues: 
This organizes all urls to be loaded in separate queues for each host.
Each host separates the crawl depth into it's own queue. The primary
rule for urls taken from any queue is, that the crawl depth is minimal.
This produces a crawl depth which is identical to the clickdepth.
Furthermorem the crawl is able to create a much better balancing over
all hosts which is fair to all hosts that are in the queue.
This process will create a very large number of files for wide crawls in
the QUEUES folder: for each host a directory, for each crawl depth a
file inside the directory. A crawl with maxdepth = 4 will be able to
create 10.000s of files. To be able to use that many file readers, it
was necessary to implement a new index data structure which opens the
file only if an access is wanted (OnDemandOpenFileIndex). The usage of
such on-demand file reader shall prevent that the number of file
pointers is over the system limit, which is usually about 10.000 open
files. Some parts of YaCy had to be adopted to handle the crawl depth
number correctly. The logging and the IndexCreateQueues servlet had to
be adopted to show the crawl queues differently, because the host name
is attached to the port on the host to differentiate between http,
https, and ftp services.
 2014-04-16 21:34:28 +02:00
Michael Peter Christen
075b6f9278 refactoring of the crawl balancer: the balancer is turned into an 
interface and the old balancer class is moved into LegacyBalancer to
make room for a fresh implementation of a crawl balancer.
 2014-04-14 13:32:35 +02:00
Michael Peter Christen
8470dfe3f8 Merge branch 'master' of ssh://git@gitorious.org/yacy/rc1.git 2014-04-14 12:17:52 +02:00
reger
46016fa153 autoupdate fails to download latest release (1.71) due to default release blacklist 
- removed the default version blacklist regex from init (for future versions)

!!!  left existing update  blacklist setting untouched !!! 
(existing installation wanting autoupdate for 1.71 need to change blacklist in ConfigUpdate_p.html)

- moved old blacklist patch to migration.java
 2014-04-13 07:32:32 +02:00
Michael Peter Christen
8aeef73d49 fix for virtual root nodes 2014-04-11 15:12:34 +02:00
Michael Peter Christen
7c7fbb9818 find depth-matches also for edge targets 2014-04-11 12:27:21 +02:00
Michael Peter Christen
dd12dd392f introduction of a data structure for HyperlinkEdges which should use 
less memory as it does no double-storage of source links for each edge
of the graph.
 2014-04-11 12:09:33 +02:00
Michael Peter Christen
6ea8bb7348 using MultiProtocolURL for edge data which is faster (hash computation 
is now much easier) and smaller in size
 2014-04-11 10:58:37 +02:00
Michael Peter Christen
b21c208b4d enhanced hashcode computation for MultiProtocolURL 2014-04-11 10:23:48 +02:00
Michael Peter Christen
ce1d1b2fa0 fix for maximum tag length in parser 2014-04-11 09:56:44 +02:00
Michael Peter Christen
17e0956312 refactoring of SystemLoad calls (only one backend tool) 2014-04-11 09:25:18 +02:00
Michael Peter Christen
a37d067692 refactoring 2014-04-10 23:46:35 +02:00
orbiter
95780eed32 Merge branch 'master' of git@gitorious.org:yacy/rc1.git 2014-04-10 21:40:54 +02:00
Michael Peter Christen
67beef657f strong redesign of html parser: object recursion is now made using a 
stack on html tag objects, not using a recursive parse-again method
which may cause bad performance and huge memory allocation. The new
method also produced better parsed image objects with exact anchor text
references.
 2014-04-10 18:58:03 +02:00
Michael Peter Christen
6bd8c6f195 fix for wrong status codes of error pages 2014-04-10 09:08:59 +02:00
Michael Peter Christen
9e503b3376 also delete the robots.txt file from the cache when a new crawl is 
started
 2014-04-09 21:59:54 +02:00
orbiter
67501c9dda Merge branch 'master' of git@gitorious.org:yacy/rc1.git 2014-04-09 19:58:54 +02:00
Michael Peter Christen
1c21b3256d fix for robots.txt handling: delete old entry before starting a new 
crawl.
 2014-04-09 18:33:48 +02:00
orbiter
c250fac9f4 linkstructure refactoring to get more options for clickdepth analysis 2014-04-09 17:52:51 +02:00
Michael Peter Christen
8068e68474 Merge branch 'master' of ssh://git@gitorious.org/yacy/rc1.git 2014-04-09 12:45:15 +02:00
Michael Peter Christen
bd886054cb new structure and enhancements for link graph computation: 
- added order option to solr queries to be able to retrieve document
lists in specific order, here: link length
- added HyperlinkEdge class which manages the link structure
- integrated the HyperlinkEdge class into clickdepth computation
- extended the linkstructure.json servlet to show also the clickdepth
and other statistic information
 2014-04-09 12:45:04 +02:00
reger
f326a67561 fix: typo in default charset in metadata2solr 
update pom and NB build to Solr 4.7.1 libs
 2014-04-06 22:31:22 +02:00
Michael Peter Christen
df138084c0 do solr optimization independently from memory and load constraints: 
- not doing an optimization will likely cause a too many files exception
- without optimization performance will be even worse which would
prevent optimization in the future as well (prevent a deadlock
situation)
 2014-04-06 11:04:23 +02:00
Michael Peter Christen
ebd44a7080 replaced solr 4.6.1 with solr 4.7.1 and added index migration to 
lucene_47
 2014-04-06 10:45:03 +02:00
Michael Peter Christen
734778c0c8 fixed a time-out problem in the default servlet which is also a logging 
problem because the error log showed the wrong reason (file not found)
instead the actual reason (time-out).
 2014-04-04 15:27:29 +02:00
Michael Peter Christen
466d90ad42 fixed a problem with resource observer; probably coming from uncatched 
exceptions within the apache library which appear only in concurrency
environments.
 2014-04-04 15:26:39 +02:00
Michael Peter Christen
e8ddd415a8 enhanced the new link structure graph 2014-04-04 14:43:54 +02:00
Michael Peter Christen
926d28dd3f fixed a bug which prevented crawl starts after a network switch 2014-04-04 14:43:35 +02:00
Michael Peter Christen
3ce8eff21b another fix for inbound/outbound detection 2014-04-04 12:41:59 +02:00
Michael Peter Christen
d4b5c457e4 NPE fix 2014-04-04 12:34:34 +02:00
Michael Peter Christen
36a66b0704 fix for parsing of numeric value in case that boolean values are given 2014-04-04 11:59:51 +02:00
orbiter
41730c8048 better logging in template engine: shows filename of servlets where 
errors in templates occur
 2014-04-04 10:55:46 +02:00
orbiter
3c1274057d fixed thread dump in case of wrong seeds 2014-04-04 10:54:56 +02:00
orbiter
18f9c40302 moved Edge class out of linkstructure servlet as this does not work on 
non-eclipse driven environments (all non-dev cases)
 2014-04-04 10:54:11 +02:00
orbiter
de95e5e524 reduced search activity corona strength in network image 2014-04-04 10:08:44 +02:00
reger
da413af664 move baseurl after parsing orig source in urlproxyservlet 
to calculate absolute href links for rewrite from unmodified source.
 2014-04-04 03:11:16 +02:00
reger
af6ad20728 fix: remove obsolete ref to yacy.home 
(use Switchboard instead)
 2014-04-04 02:45:04 +02:00
Michael Peter Christen
74ab094587 fix for solr query size; too many documents had been retrieved in case 
that less than _pagesize_ had been requested.
 2014-04-03 13:42:10 +02:00
Michael Peter Christen
c64c10ef00 Merge branch 'master' of ssh://git@gitorious.org/yacy/rc1.git 2014-04-03 01:58:06 +02:00
Michael Peter Christen
48fbfa60c1 bugfix to inbound/outbound identification 2014-04-03 01:21:43 +02:00
reger
227c42bc96 eleminate obsolete URIMetaDataRow class 
by joining it with/into URIMetaDataNode.
 2014-04-03 00:35:15 +02:00
Michael Peter Christen
cca851a417 introduced new solr field crawldepth_i which records the crawl depth of 
a document. This is the upper limit for the clickdepth_i value which may
be shorter in case that the crawler did not take the shortest path to
the document.
 2014-04-02 23:37:01 +02:00
orbiter
b1ba764d81 fix for first start options and added german translation for popup texts 2014-04-02 17:10:59 +02:00
orbiter
429a874222 - added COLS field in GSA response (non-gsa standard by customer 
request)
- updated document link in GSA response writer
 2014-04-02 16:05:44 +02:00
Michael Peter Christen
1b9ec9a1c5 - added popover to p2p/stealth mode button to explain the peer mode and 
privacy issues.
- added popover to first-time use case to explain that specific servlets
are only visible after customization and/or crawl starts
 2014-04-02 13:33:43 +02:00
Michael Peter Christen
62a36fa584 Merge branch 'master' of ssh://git@gitorious.org/yacy/rc1.git 2014-04-02 03:27:08 +02:00
reger
c9f92abddc fix: application link count 
(URIMetadataNode)
 2014-04-02 03:21:51 +02:00
Michael Peter Christen
a267c46e1a Merge branch 'master' of ssh://git@gitorious.org/yacy/rc1.git 2014-04-02 02:35:58 +02:00
Michael Peter Christen
5b83887da8 npe fix 2014-04-02 02:34:55 +02:00
Michael Peter Christen
63c9fcf3e0 free configuration of postprocessing clickdepth maximum depth and time 2014-04-02 02:34:39 +02:00
Michael Peter Christen
39b641d6cd added tutorial mode - some menu items will only appear if you 'qualify' 
for them. Thus, the first-time user will only see four menu items. The
other items will unfold as the user interacts.
 2014-04-02 02:33:17 +02:00
sixcooler
f06775850f fix receiving DHT / parse pultipart 
+ another close to fix possible resource leak warning
 2014-04-02 01:24:15 +02:00
reger
49e76a1c55 make use of detected charset in htmlParser if none is given. 2014-04-01 04:02:34 +02:00
reger
e11504309f adding a hint to javascript browser short cut on Url-Proxy page (AugmentedBrowsing_p.html) 2014-03-30 05:11:42 +02:00
reger
b12200cafe alternative UrlProxyServlet (for /proxy.html) using different url rewrite rules 
- use JSoup parser for selective rewrite of html body <a href=  links only,
instead of regex which rewrites also header href/src links
- this improves display of pages which use header <base> tag
- tags with src attribute are taken from original location (like css) improving display and are not routed trough the indexer
Disadvantage: scripting links will drop out of proxy

Setting of the servlet through web.xml exclusivly (in case one would like to quickly switch back to the YaCyProxyServlet,
leaving the existing code of YaCyProxyServlet untouched available)
 2014-03-30 04:04:02 +02:00
reger
2953ebe701 fix: port in local target adress 
& button style
 2014-03-29 00:34:01 +01:00
Michael Peter Christen
fda591695c fixed visibility of custom icon 2014-03-28 17:25:39 +01:00
Michael Peter Christen
a9b9950d7f Merge branch 'master' of ssh://git@gitorious.org/yacy/rc1.git 2014-03-28 14:48:32 +01:00
Michael Peter Christen
b488f33975 added close to fix possible resource leak warning 2014-03-28 14:34:49 +01:00
Michael Peter Christen
56710ecb26 prevent opening of new files as that could be a cause for the latest 
too-many-open-files exception. The old file is just truncated if the
table is cleaned.
 2014-03-28 14:31:43 +01:00
Michael Peter Christen
8b44fcf0f4 added missing @Override annotation 2014-03-28 13:48:37 +01:00
reger
d7055904a6 fix: proxyservlet path header setting 2014-03-28 02:05:58 +01:00
Michael Peter Christen
e515dd460d added linkscount_i and linksnofollowcount_i to the default solr schema 2014-03-27 23:36:08 +01:00
Michael Peter Christen
1a764135be one more Thread Dump fix for new bootstrap css style 2014-03-27 23:01:28 +01:00
Michael Peter Christen
bb21d825f9 fix for thread dump line spacing 2014-03-27 22:13:37 +01:00
Michael Peter Christen
cbdfef7ce1 changed protocol facet to show also all other counts if one facet is 
selected
 2014-03-27 13:29:14 +01:00
reger
b9056ef2db remove unused private header entries (HeaderFramework) 
X_YACY_ORIGINAL_REQUEST_LINE
X_YACY_KEEP_ALIVE_REQUEST_COUNT
CONNECTION_PROP_REQUESTLINE
 2014-03-26 23:28:19 +01:00
sixcooler
6d16fa993d make transparent proxy handle https-connections: 
the implemented handle for connect did not work for me - so lets try the
connectHandler
 2014-03-26 20:01:15 +01:00
Michael Peter Christen
61ad194065 fix for source and target clickdepth in webgraph index 2014-03-26 16:00:05 +01:00
Marc Nause
809b4e1fd9 Team added support for URLs with unicode characters in host part to 
blacklist. Punycode is used to handle unicode characters.
 2014-03-25 22:14:54 +01:00
reger
b126b9ba17 add some InputFileStream close at end of reads 
to make sure file is released
 2014-03-24 02:32:17 +01:00
reger
ca7444dbdf limit filetype nav to known extension also on image/media search 
- on text search we limit filetype nav already to known extension, apply filter to image search
 2014-03-23 23:10:29 +01:00
reger
651d057e93 surrogate import translate dc:language 3-char codes 
OAI records often use 3-char language codes, start converting some 3-char lang's to the internal ISO639-1 2-char code
 2014-03-23 00:40:36 +01:00
orbiter
22618e3ba2 Merge branch 'master' of ssh://git@gitorious.org/yacy/rc1.git 2014-03-21 20:28:50 +01:00