Commit Graph

297 Commits

Author SHA1 Message Date
Michael Peter Christen
c00efc2717 made the solr connection more generic 2012-05-09 16:46:45 +02:00
Michael Peter Christen
ba6aaabc51 refactoring + parser bugfixes 2012-05-04 17:28:27 +02:00
Michael Peter Christen
942896fe46 removed methods not supported by new solrj connector for httpclient 4
Error was:
java.lang.UnsupportedOperationException: Client was created outside of
HttpSolrServer
	at
org.apache.solr.client.solrj.impl.HttpSolrServer.setDefaultMaxConnectionsPerHost(HttpSolrServer.java:614)
	at
net.yacy.cora.services.federated.solr.SolrSingleConnector.<init>(SolrSingleConnector.java:128)
	at
net.yacy.cora.services.federated.solr.SolrShardingConnector.<init>(SolrShardingConnector.java:55)
	at net.yacy.search.Switchboard.<init>(Switchboard.java:657)
	at net.yacy.yacy.startup(yacy.java:222)
	at net.yacy.yacy.main(yacy.java:1018)
2012-04-27 18:26:36 +02:00
Michael Peter Christen
22e1f68c0b solrj user authentication patch 2012-04-27 17:53:45 +02:00
Michael Peter Christen
62f2554a01 - fixed build problems (deprecated methods using httpclient 3.1)
- removed httpclient 3.1 lib which was used by solrj (solrj now uses
httpclient 4)
2012-04-27 17:46:08 +02:00
Michael Peter Christen
a6d60fc21f concurrency enhancement in ConfigurationSet 2012-04-27 17:20:18 +02:00
Michael Peter Christen
453010bd68 - solved problems with backpath normalization
- redesigned in/outbound link handover
- removed iframe links from inbound/outbound in solr scheme
2012-04-27 16:48:51 +02:00
Michael Peter Christen
7860c1df80 fix needed for new solrj library 2012-04-27 14:13:59 +02:00
Michael Peter Christen
0e13022147 - enhanced solr field documentation
- added xml api button to IndexFederated_p - the solr schema.xml file
can be generated by YaCy
2012-04-26 15:25:07 +02:00
Michael Peter Christen
f8cd57c92f new indexing strategy: ALL links that appear anywhere are indexed, not
only links where the content can be parsed. All non-parseable links are
placed into the noload queue. The search process must therefore be able
to filter out non-text search results.
- This fixes the problem that image search results appeared in the text
search.
- The interactive search can retrieve now ALL types of links
- The p2p interface is now extended to retrieve only certain types of
links (text, image, video, apps)
- The search process has an extension to filter the right document type
according to the search query
2012-04-22 02:05:17 +02:00
Michael Peter Christen
14f67f217c refactoring of ContentDomain: now subclass of Classification 2012-04-22 00:04:36 +02:00
Michael Peter Christen
8a08c96a82 removed dependency from logging 2012-04-21 21:32:31 +02:00
Michael Peter Christen
a1a5b015d8 refactoring: moved document Classification to cora package 2012-04-21 21:31:13 +02:00
Michael Peter Christen
898fa7c3f3 use tld heuristic to check if a domain is local or global 2012-02-25 15:41:20 +01:00
Michael Peter Christen
665626a51b catch OOM errors during scanning 2012-02-24 02:15:27 +01:00
Michael Peter Christen
33a405dab8 ipv6 bugfix 2012-02-24 00:50:46 +01:00
Michael Peter Christen
c6c61be3f0 fix for http://bugs.yacy.net/view.php?id=148 2012-02-24 00:38:57 +01:00
Michael Peter Christen
e101c2e0e2 added changes from copperdust (submitted by email):
1. Improved and fixed language detection:
	1.1 Identificator.java - recognition fix (improved)
	1.2 DCEntry.java - fix (changed detection order due to detection from
tld in many cases is incorrect)
	1.3 MultiProtocolURI.java - fixed and enhanced language from tld
detection (all currently used top-level domains; ccTLD added but not
tested).
2. Ukrainian language update.
3. Main Slavic languages langstats (tested and works fine).
2012-02-22 12:21:27 +01:00
Michael Peter Christen
8b0920b0b5 tried to fix the ipv6 problem as reported in bug
but this did not solve all problems because a bug in the apache http
client prevented that it worked. Thread dump:
Caused by: java.lang.NumberFormatException: For input string:
"1450:400c:c01:0:0:0:69"
	at
java.lang.NumberFormatException.forInputString(NumberFormatException.java:48)
	at java.lang.Integer.parseInt(Integer.java:458)
	at java.lang.Integer.parseInt(Integer.java:499)
	at org.apache.http.client.utils.URIUtils.extractHost(URIUtils.java:310)
	at
org.apache.http.impl.client.AbstractHttpClient.determineTarget(AbstractHttpClient.java:764)
	at
org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:754)
	at net.yacy.cora.protocol.http.HTTPClient.execute(HTTPClient.java:597)
	at
net.yacy.cora.protocol.http.HTTPClient.getContentBytes(HTTPClient.java:558)
	at net.yacy.cora.protocol.http.HTTPClient.GETbytes(HTTPClient.java:341)
	at de.anomic.crawler.retrieval.HTTPLoader.load(HTTPLoader.java:131)
	at de.anomic.crawler.retrieval.HTTPLoader.load(HTTPLoader.java:74)
	at
net.yacy.repository.LoaderDispatcher.loadInternal(LoaderDispatcher.java:274)
	at net.yacy.repository.LoaderDispatcher.load(LoaderDispatcher.java:164)
	at net.yacy.repository.LoaderDispatcher.load(LoaderDispatcher.java:150)
	at
net.yacy.repository.LoaderDispatcher.loadDocument(LoaderDispatcher.java:355)
	at getpageinfo_p.respond(getpageinfo_p.java:97)
2012-02-01 22:26:19 +01:00
Michael Peter Christen
ed39ef2890 changed generation of protocol information 2012-02-01 18:12:59 +01:00
Michael Peter Christen
8bee1472c9 there is no noindex, only nofollow in links 2012-01-31 23:46:35 +01:00
Michael Peter Christen
3cd6dcd352 do not add new solr fields as activated fields 2012-01-31 22:21:48 +01:00
Michael Peter Christen
987b412491 updated solr scheme: generic declaration of solr schemes 2012-01-13 11:25:15 +01:00
Marek Otahal
1dc5d9f0f3 make ConnectionInfo comparable and sort list of connections in Connections_p
ConnectionInfo compare by initTime
Connections_p implement wish to sort connections, descending

Signed-off-by: Marek Otahal <markotahal@gmail.com>
2012-01-09 01:14:41 +01:00
Roland 'Quix0r' Haeder
fa08ed5ae5 Fixed a lot CHMOD rights (no need for execute flag on *.java/*.html) and introduced local/remote crawl size ratio based check 2011-12-29 00:33:16 +01:00
Michael Christen
ee9aae5cc0 more about CreativeCommons license vocabulary 2011-12-18 16:07:51 +01:00
Michael Christen
c75e1a3125 Merge branch 'master' of ssh://git@gitorious.org/yacy/rc1.git 2011-12-18 08:20:10 +01:00
Michael Christen
13f5b5f80d the component part in the YaCy Metadata is filled using the Dubling Core
vocabulary
2011-12-18 08:19:48 +01:00
Michael Peter Christen
8d2cbfb685 more vocabularies and more semantics for lod data structures 2011-12-18 08:12:34 +01:00
Michael Christen
9cd36b4c44 added vocabulary for geolocalization as used in georss 2011-12-17 02:03:45 +01:00
Michael Christen
66ab51f89d added rdf vocabulary 2011-12-17 01:09:16 +01:00
Michael Christen
c04bfaa51b refactoring 2011-12-16 23:59:29 +01:00
Michael Peter Christen
136b514f52 added a Triple Store based on Nodes that fit to the new storage classes.
Added also a first Vocabulary for the node store - Dublin Core.
2011-12-16 23:01:47 +01:00
Michael Christen
1f4afb4dc0 performance hacks 2011-12-15 15:15:53 +01:00
Michael Peter Christen
4243ace863 added phonetic classes 2011-12-14 17:33:18 +01:00
Michael Christen
204c29f010 small bugfixes for search result display and cache display 2011-12-10 01:35:38 +01:00
orbiter
35a9e8f307 - fixed network graphic
- debuged evaluation tables
- changed cache settings in template engine
- some speed hacks
- changed int angles for peer positions in network graphic to double angles

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@8124 6c8d7289-2bf4-0310-a012-ef5d649a1542
2011-11-30 20:50:41 +00:00
orbiter
e22f8497c9 - tested the ARC methods
- removed strict authentication (if password is empty; this was buggy and not useful; can be switched on if necessary globally and not for each interface method)
- increased speed of CrawlResults page (no dns lookup any more)
- increased speed of favicon display (removed dns lookup)

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@8104 6c8d7289-2bf4-0310-a012-ef5d649a1542
2011-11-25 14:09:25 +00:00
orbiter
5a55397f99 some last-minute performance hacks
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@8101 6c8d7289-2bf4-0310-a012-ef5d649a1542
2011-11-25 11:23:52 +00:00
orbiter
804e48888b smaller bug fixes for search behavior; should produce less unnecessary removals and an exact number of results as shown in counter
should also be a little bit faster

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@8057 6c8d7289-2bf4-0310-a012-ef5d649a1542
2011-11-18 13:09:07 +00:00
orbiter
78ce3b13be typo
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@8027 6c8d7289-2bf4-0310-a012-ef5d649a1542
2011-11-10 11:57:26 +00:00
orbiter
0d858d48ec replaced String with StringBuilder in suggestion process
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@8020 6c8d7289-2bf4-0310-a012-ef5d649a1542
2011-11-09 14:42:55 +00:00
orbiter
37e35f2741 normalization of url using urlencoding/decoding
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@8017 6c8d7289-2bf4-0310-a012-ef5d649a1542
2011-11-08 12:02:22 +00:00
orbiter
e58438c01c - added a new retry connector for solr (for cases where solr responses are slow)
- added a new exist property into the metadataRepository which includes solr entries

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@8016 6c8d7289-2bf4-0310-a012-ef5d649a1542
2011-11-08 11:49:04 +00:00
orbiter
94eab08794 - updated opensearchdescription text and icon
- removed automatic setting of maxitems during search (can be set now elsewhere)
- updated RSSMessage.java

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@8009 6c8d7289-2bf4-0310-a012-ef5d649a1542
2011-10-30 01:09:38 +00:00
orbiter
a9838f8b99 fix for http://bugs.yacy.net/view.php?id=59
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7997 6c8d7289-2bf4-0310-a012-ef5d649a1542
2011-10-12 22:26:48 +00:00
orbiter
1b45e33f04 added robots tag parser to solr scheme
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7986 6c8d7289-2bf4-0310-a012-ef5d649a1542
2011-09-30 13:39:01 +00:00
orbiter
cf4fd525ee added directDocByURL attribute in crawl profile
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7985 6c8d7289-2bf4-0310-a012-ef5d649a1542
2011-09-30 12:38:28 +00:00
orbiter
813f297a95 another performance hack: re-use of known host addresses for isLocal property; avoids look-up in local hash
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7983 6c8d7289-2bf4-0310-a012-ef5d649a1542
2011-09-30 08:26:31 +00:00
orbiter
b250e6466d implemented crawl restrictions for IP pattern and country lists
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7980 6c8d7289-2bf4-0310-a012-ef5d649a1542
2011-09-29 15:17:39 +00:00