Commit Graph

239 Commits

Author SHA1 Message Date
apfelmaennchen
a425fbd8d6 - created new target 'portalsearch' in build.xml to generate yacy-portalsearch.tar.gz for static hosting
- some refactoring for search widget and jquery
- update for ConfigLiveSearch.html to refelct latest changes

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@8023 6c8d7289-2bf4-0310-a012-ef5d649a1542
2011-11-09 21:01:38 +00:00
orbiter
3f606407bc added new scripts to bin in build
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7991 6c8d7289-2bf4-0310-a012-ef5d649a1542
2011-10-07 22:57:20 +00:00
orbiter
d2ea250d99 refactoring:
- moved many classes from de.anomic to net.yacy
- made more sub-packages for search classes

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7973 6c8d7289-2bf4-0310-a012-ef5d649a1542
2011-09-25 16:59:06 +00:00
orbiter
65ab067491 migration to solrj 3.4.0
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7952 6c8d7289-2bf4-0310-a012-ef5d649a1542
2011-09-14 20:08:59 +00:00
orbiter
dc25c48fc9 added more libraries that are needed by solrj
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7922 6c8d7289-2bf4-0310-a012-ef5d649a1542
2011-09-02 23:43:04 +00:00
sixcooler
52b477cf6f bump to httpclient-4.1.2, httpcore-4.1.3 - bugfixrelease
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7876 6c8d7289-2bf4-0310-a012-ef5d649a1542
2011-08-12 17:42:32 +00:00
sixcooler
48560a44a9 bump to httpcore-4.1.2: a bugfixrelease
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7853 6c8d7289-2bf4-0310-a012-ef5d649a1542
2011-07-25 00:48:29 +00:00
orbiter
d3c89b90ce temporary adding the old httpclient-3.1 again because the solrj classes need them. should be removed as soon solrj supports httpclient-4
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7831 6c8d7289-2bf4-0310-a012-ef5d649a1542
2011-07-04 17:04:49 +00:00
orbiter
768c59740c - replaced solrj 3.1 with solrj 3.3
- updated also slf4j
- added authentication for solrj


git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7829 6c8d7289-2bf4-0310-a012-ef5d649a1542
2011-07-04 16:35:30 +00:00
orbiter
528b59e078 replaced xerces.jar library that was originally added 2005 with SVN 126 to the libx directory and that was moved to lib in SVN 5781
the new replacement is taken from http://xerces.apache.org and has the version 2.11.0 and was inside the file Xerces-J-bin.2.11.0.tar.gz
and consists of two files named xercesImpl.jar and xml-apis.jar
The original purpose of that library was to support:
- content parsers
- optional seed uploader
- SOAP API (which will be committed later)
Since the SOAP API does not exist any more the purpose is to support content parser and an optional seed uploader

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7819 6c8d7289-2bf4-0310-a012-ef5d649a1542
2011-07-02 22:33:35 +00:00
orbiter
e7e1a0f328 replaced commons-io v1.4 with v2.0.1
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7818 6c8d7289-2bf4-0310-a012-ef5d649a1542
2011-07-02 21:10:13 +00:00
orbiter
5092a14bcb replaced fontbox, jempbox, pdfbox v 1.5 with v1.6
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7817 6c8d7289-2bf4-0310-a012-ef5d649a1542
2011-07-02 20:52:33 +00:00
suessthomas
ccad615f58 The Java-XMS and Xmx values for the target of "run" (run YaCy) inserted.
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7777 6c8d7289-2bf4-0310-a012-ef5d649a1542
2011-06-11 21:22:08 +00:00
orbiter
77fe69395d added jempbox-1.5.0.jar which is required by pdfbox-1.5 as stated in http://pdfbox.apache.org/dependencies.html
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7774 6c8d7289-2bf4-0310-a012-ef5d649a1542
2011-06-05 20:04:41 +00:00
sixcooler
efcd21e0ed new httpclient, httcore (bugfixrelease)
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7769 6c8d7289-2bf4-0310-a012-ef5d649a1542
2011-06-02 21:34:50 +00:00
orbiter
761b1c71dc added latest pdfbox
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7761 6c8d7289-2bf4-0310-a012-ef5d649a1542
2011-05-30 14:56:36 +00:00
orbiter
3b578a28ef some patches to prevent that empty or bad IP information is broadcasted
- on client-side: fix bad IP reports from remote Peers by replacing their reported IP with their server IP if the reported IP is bad, broken or disallowed
- on server-side: the same during a peer ping (here the ping'ed server acts also as client during the back-ping) and also when receiving a message or a search where the client sends also its seed. Here the IP is replaced by the client IP if the reported IP is broken or bad

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7687 6c8d7289-2bf4-0310-a012-ef5d649a1542
2011-04-29 10:58:12 +00:00
orbiter
c493f101c0 added one more script file to release build script
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7681 6c8d7289-2bf4-0310-a012-ef5d649a1542
2011-04-28 13:19:24 +00:00
orbiter
f6077b3cc0 added more attributes for html parser and enhanced data structures
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7679 6c8d7289-2bf4-0310-a012-ef5d649a1542
2011-04-28 13:09:01 +00:00
apfelmaennchen
a0e4960a4d YMark:
- first attempt for a firefox json bookmark importer
- added JSON library json-simple-1.1.jar

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7658 6c8d7289-2bf4-0310-a012-ef5d649a1542
2011-04-15 20:58:58 +00:00
orbiter
19fd13d3bc Added federated index storage to solr.
YaCy supports now the storage to remote solr indexes.
More federated storage (and search) methods may follow.

The remote index scheme is the same as produced by the SolrCell; see
http://wiki.apache.org/solr/ExtractingRequestHandler
Because this default scheme is used, the default example scheme can be used as solr configuration
This is also the same scheme that solr uses if documents are imported with apache tika.

federated solr storage is switched off by default.

To use this, do the following:
- set federated.service.solr.indexing.enabled = true
- download solr from http://www.apache.org/dyn/closer.cgi/lucene/solr/
- extract the solr (3.1) package, 'cd example' and start solr with 'java -jar start.jar'
- start yacy and then start a crawler. The crawler will fill both, YaCy and solr indexes.
- to check whats in solr after indexing, open http://localhost:8983/solr/admin/

Until now it is not possible to use the solr index to search with YaCy in that solr index.
This functionality is now available for two reasons:
1) to compare the functionality of Solr and YaCy and to compare the search speed
2) to use YaCy as a search appliance for people who need a crawler or other source harvesting methods
   that YaCy provides (like dublin core reading, wikimedia dump reading, rss feed reader etc) if people still
   want to use solr instead of YaCy.

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7654 6c8d7289-2bf4-0310-a012-ef5d649a1542
2011-04-14 20:05:04 +00:00
orbiter
4c013d9088 more UTF8 getBytes() performance hacks
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7649 6c8d7289-2bf4-0310-a012-ef5d649a1542
2011-04-12 05:02:36 +00:00
f1ori
399d7d6878 * fix permissions of bin/-folder in debian package
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7647 6c8d7289-2bf4-0310-a012-ef5d649a1542
2011-04-07 07:31:17 +00:00
f1ori
21fe5e6c6a * add bin-folder to debian package
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7638 6c8d7289-2bf4-0310-a012-ef5d649a1542
2011-04-02 10:58:56 +00:00
orbiter
18ec7fe53c added a clearall.sh script that deletes the complete index and everything else that belongs to crawling
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7611 6c8d7289-2bf4-0310-a012-ef5d649a1542
2011-03-20 08:36:29 +00:00
orbiter
d98884f1d5 added script for importmediawiki.sh in build.xml
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7610 6c8d7289-2bf4-0310-a012-ef5d649a1542
2011-03-19 23:58:11 +00:00
suessthomas
9956dc9dce Update jcifs-library to Version 1.3.15. Small Changes, read: http://jcifs.samba.org/ - "Minor adjustments have been applied to DcerpcHandle locking routines in the SID class to fix sporadic occurances of "All pipe instances are busy" errors under high load."
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7587 6c8d7289-2bf4-0310-a012-ef5d649a1542
2011-03-13 20:09:52 +00:00
sixcooler
45dcfa3460 update to httpclient-4.1
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7473 6c8d7289-2bf4-0310-a012-ef5d649a1542
2011-02-08 21:46:24 +00:00
f1ori
d38c739b31 * add ant-target deleteData, to simulate fresh yacy installation
* useful in IDEs like eclipse

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7472 6c8d7289-2bf4-0310-a012-ef5d649a1542
2011-02-08 09:12:48 +00:00
orbiter
19711ff6d6 added target that compiles only the cora package
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7423 6c8d7289-2bf4-0310-a012-ef5d649a1542
2011-01-04 00:44:03 +00:00
lotus
2db6ca8104 - do not allow user to disable java installation
- java version update
- build cleanup

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7284 6c8d7289-2bf4-0310-a012-ef5d649a1542
2010-10-30 08:49:39 +00:00
orbiter
fc3ee9d8a1 changed paths for new libs also in Info.plist
when changing paths in build.xml, please also update the paths in /addon/YaCy.app/Contents/Info.plist

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7198 6c8d7289-2bf4-0310-a012-ef5d649a1542
2010-09-27 14:49:51 +00:00
sixcooler
f4357dff03 bump to httpclient-4.0.3 which fixes a number of bugs
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7197 6c8d7289-2bf4-0310-a012-ef5d649a1542
2010-09-27 13:24:40 +00:00
f1ori
e670e1ef8e add charset auto-detection for htmlParser
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7186 6c8d7289-2bf4-0310-a012-ef5d649a1542
2010-09-25 18:02:21 +00:00
orbiter
3552476fbe terminated migration from apache httpclient-3.1 to 4.1:
- remove the library
- added two classes from the httpclient-3.1 library as source code to YaCy because these classes were used by the YaCy HTTP Server
- modified the added classes ChunkedInputStream and ContentLengthInputStream in such a way that:
 * there are no more dependencies to httpclient-3.1
 * these classes had been simplified to serve only the purpose for the YaCy httpd

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7171 6c8d7289-2bf4-0310-a012-ef5d649a1542
2010-09-20 08:36:48 +00:00
orbiter
f9a27a05e5 migrated to log4j 1.2.16
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7153 6c8d7289-2bf4-0310-a012-ef5d649a1542
2010-09-15 09:18:35 +00:00
orbiter
5c67e6ca49 migrated to latest apache commons fileupload 1.2.2
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7152 6c8d7289-2bf4-0310-a012-ef5d649a1542
2010-09-15 08:54:41 +00:00
orbiter
5fe828fa06 - replaced pdfbox and fontbox version 1.1.0 with 1.2.1
- added some clear statements that shall clear static cache size within the pdfbox library
- the pdfbox library contains a memory leak; it is unsafe to run a peer with pdf parser permanently on.

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7120 6c8d7289-2bf4-0310-a012-ef5d649a1542
2010-09-07 17:13:47 +00:00
orbiter
fa5683adfe create a mac dmg file (a disc image) for mac releases in ant
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7091 6c8d7289-2bf4-0310-a012-ef5d649a1542
2010-09-02 19:11:49 +00:00
lotus
091281c9f2 Mac app ant task building a ready-to-distribute zip file
extending r7080

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7089 6c8d7289-2bf4-0310-a012-ef5d649a1542
2010-09-02 08:01:01 +00:00
lotus
e9160ea1e5 Mac ant task according to r7023
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7080 6c8d7289-2bf4-0310-a012-ef5d649a1542
2010-08-27 18:40:32 +00:00
orbiter
189a986ebd - modified api-call interface to record api calls with references to api-call database (carries pk)
- added recording date, last execution date and next execution date for a scheduler (scheduler to be implemented next)
- extended database access methods for more data formats, especially for date insert/retrieval
- extended 'Steering' interface to show new database fields
- migrated Steering to new http client
- extended cora http client to transmit authentication and also added some convenience methods (http response code)
- simplified database back-end (not so much specialized methods for multiple properties)
- extended date formatter to produce a special format to show dates in html (  in spaces of date format)


git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7049 6c8d7289-2bf4-0310-a012-ef5d649a1542
2010-08-18 15:56:38 +00:00
mikeworks
6b13101d18 Collage.html: Fixed problem where the German translation broke the action that contained Collage in a form
build.xml: Fixed check for existing private.key, added check for non existing release in target sign and changed the include filenames for changed libs
Added log4j.properties file to eliminate the warning about a not initialized log4j subsystem with parameters for one console appender

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6998 6c8d7289-2bf4-0310-a012-ef5d649a1542
2010-07-26 20:18:19 +00:00
sixcooler
c5c67f0504 start migrating to HttpComponents-Client-4.x
see http://forum.yacy-websuche.de/viewtopic.php?f=5&t=2872

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6965 6c8d7289-2bf4-0310-a012-ef5d649a1542
2010-07-12 23:07:05 +00:00
orbiter
b5e190099d - updated pdfbox and fontbox to 1.1.0
- added license file to sbbi-upnplib

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6946 6c8d7289-2bf4-0310-a012-ef5d649a1542
2010-06-26 10:58:07 +00:00
orbiter
37b8827a7a - removed the UPnP library sources from sbbi and added the jar library again. The library was included to get support for fedora releases, but after this time the fact that the sbbi cannot be part of fedora should be re-discussed. If this will still not be possible, then we may integrate the sbbi UPnP package using reflection.
- cleaned uo the code. The new eclipse helios provided new warnings for dead code. This change cleans up most of these warnings

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6945 6c8d7289-2bf4-0310-a012-ef5d649a1542
2010-06-26 10:32:47 +00:00
orbiter
89b4fff1c2 adopted ant script for new exif library
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6746 6c8d7289-2bf4-0310-a012-ef5d649a1542
2010-03-12 12:36:38 +00:00
orbiter
1bbe14d23f SVN 6716 unfortunately contained parts of the unfinished SMB integration. To fix compile errors the remaining parts of the SMB implementation stub is added with this commit.
This adds the jcifs smb library.

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6717 6c8d7289-2bf4-0310-a012-ef5d649a1542
2010-03-05 21:46:22 +00:00
orbiter
6fde481ef4 missing for last commit
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6692 6c8d7289-2bf4-0310-a012-ef5d649a1542
2010-02-22 20:21:39 +00:00
orbiter
1e2c011c98 updated the jsch lib from 0.1.21 to 0.1.42
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6688 6c8d7289-2bf4-0310-a012-ef5d649a1542
2010-02-21 23:43:50 +00:00