Marown/yacy_search_server
1
0
Fork 0
mirror of https://github.com/yacy/yacy_search_server.git synced 2024-09-21 00:00:13 +02:00
Code Issues Actions Packages Projects Releases Wiki Activity
4,466 Commits 1 Branch 20 Tags 836 MiB
ae03a54d23
Commit Graph

4466 Commits

This Branch
This Branch
All Branches
Author SHA1 Message Date
danielr
ae03a54d23 pdfParser: updated lib, fixed ClassNotFoundException: CMSError 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4776 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-05-08 16:55:45 +00:00
danielr
0d3808bd9e minor refactoring 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4775 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-05-08 16:51:01 +00:00
orbiter
719f5defb1 updated some grafics at new terminal_p 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4774 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-05-07 23:42:14 +00:00
orbiter
58830e9b28 added new terminal servlet using current visualization methods and a new one: a processing (processing.org) applet. 
the new servlet can be found at
http://localhost:8080/terminal_p.html
..to be enhanced..

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4773 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-05-07 22:49:07 +00:00
lotus
9bc56a9edc xss protection 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4772 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-05-07 16:37:13 +00:00
orbiter
b32736762c enhanced rssTerminal 
- 3 lines possible
- distinguishing of private and public data, if not authorized only public data is shown
- shows now more events, including local searches in clear text if user is logged in
- simplyfied peer events
- better recognition of 'real' new peers
- presentation of peer pings from other peers

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4771 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-05-06 23:05:48 +00:00
orbiter
fbb712c669 refactoring: 
moved importer classes to crawler and plasma package

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4770 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-05-06 13:44:38 +00:00
orbiter
ee81ff4ef4 added crawler target directory for build 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4769 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-05-06 00:37:51 +00:00
orbiter
1689030ee8 refactoring: moved all crawler classes into their own package 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4768 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-05-06 00:32:41 +00:00
orbiter
fe4871ac02 removed empty package 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4767 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-05-05 23:54:00 +00:00
orbiter
3082edfdbc ups 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4766 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-05-05 23:19:49 +00:00
orbiter
d2ba1fd2ab major step forward to network switching (target is easy switch to intranet or other networks .. and back) 
This change is inspired by the need to see a network connected to the index it creates in a indexing team.
It is not possible to divide the network and the index. Therefore all control files for the network was moved to the network within the INDEX/<network-name> subfolder.
The remaining YACYDB is superfluous and can be deleted.
The yacyDB and yacyNews data structures are now part of plasmaWordIndex. Therefore all methods, using static access to yacySeedDB had to be rewritten. A special problem had been all the port forwarding methods which had been tightly mixed with seed construction. It was not possible to move the port forwarding functions to the place, meaning and usage of plasmaWordIndex. Therefore the port forwarding had been deleted (I guess nobody used it and it can be simulated by methods outside of YaCy).
The mySeed.txt is automatically moved to the current network position. A new effect causes that every network will create a different local seed file, which is ok, since the seed identifies the peer only against the network (it is the purpose of the seed hash to give a peer a location within the DHT).
No other functional change has been made. The next steps to enable network switcing are:
- shift of crawler tables from PLASMADB into the network (crawls are also network-specific)
- possibly shift of plasmaWordIndex code into yacy package (index management is network-specific)
- servlet to switch networks 

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4765 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-05-05 23:13:47 +00:00
danielr
d70a472460 added file for previous commit 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4764 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-05-05 05:19:01 +00:00
danielr
d32fe84472 added default User-Agent 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4763 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-05-04 17:26:19 +00:00
danielr
8c5f062e0b corrected YaCy version in HTTP User-Agent 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4762 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-05-04 12:18:00 +00:00
danielr
d7b21bc90c re-added gzip POST for transferRWI/URL (HTTP/1.1 compliant) 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4761 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-05-04 10:53:04 +00:00
f1ori
a5a1f19368 * allow to force login for xbel, needed for yacybar 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4757 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-05-03 11:13:27 +00:00
low012
8d83febb95 *) BlacklistCleaner_p.java reports exception to log instead of System.err 
*) changes in formatting for better readability in BlacklistCleaner_p.java
*) replaced  test for necessary Java version (was 1.4.2, is 1.5 now)

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4756 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-05-03 10:16:04 +00:00
danielr
d4bce6affd refactoring (initialized static fields, removed empty if/else, serialized some fields in serializable classes) 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4755 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-05-03 09:06:00 +00:00
daburna
19ca452666 updated language file 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4754 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-05-02 20:19:05 +00:00
danielr
be2c9c07ff escape some unescaped characers in URLs (fixes problems with proxy) 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4753 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-05-02 08:19:47 +00:00
orbiter
d0678f7ab9 refactoring as result of 
http://forum.yacy-websuche.de/viewtopic.php?f=6&t=959&p=7560#p7560

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4752 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-05-01 22:40:42 +00:00
orbiter
483e9a2066 - shifted tld recognition methods from yacyURL to serverDomains 
- changed isLocal Property in such a way that it is possible to see if a domain is in the internet (and not intranet)

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4751 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-04-30 23:06:42 +00:00
orbiter
a3df23659c re-implementation of charset checking 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4750 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-04-30 13:23:05 +00:00
orbiter
75a1702133 - fix for ConcurrentModificationException during shutdown 
- fix for Ranking distribution problem (suma-lab peer does not exist any more)

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4749 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-04-30 11:19:52 +00:00
orbiter
27ab0a5f89 fixed XSS problem in ConfigProperties 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4748 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-04-29 22:47:00 +00:00
orbiter
32b5b057b9 - modified, simplified old kelondroHTCache object; I believe it should be replaced by something completely new 
- removed tree data type in kelondroHTCache
- added new class kelondroHeap; may be the core for a storage object that will once replace the many-files strategy of kelondroHTCache
- removed compatibility mode in indexRAMRI


git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4747 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-04-29 22:31:05 +00:00
orbiter
d3715e02ae removed double/redundant servlet Config_p 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4746 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-04-29 19:19:14 +00:00
danielr
ec84a52adb change for problem with NPE (seen as "PROXY Unknown Error while processing request") 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4745 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-04-29 16:06:54 +00:00
orbiter
5813cc149f fix for bad rssTerminal behavior 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4744 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-04-28 20:34:37 +00:00
orbiter
88216c1f1f fix for http://forum.yacy-websuche.de/viewtopic.php?f=6&t=1103&hilit=&p=7362#p7362 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4743 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-04-26 22:59:20 +00:00
orbiter
d0b893523e - protection against RAM overflow caused by new peer rss news 
- more XSS protection

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4742 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-04-26 22:53:04 +00:00
orbiter
685794e7e7 fix for parser/encoding Exception 
see http://forum.yacy-websuche.de/viewtopic.php?f=5&t=1111&hilit=&sid=55a320b54e1e3bda9410e7c50b5147f1&p=7431#p7431

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4741 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-04-26 22:14:45 +00:00
orbiter
cf042e6957 reverted change by mistake in yacyVersion 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4740 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-04-26 01:08:59 +00:00
orbiter
9935e83c86 added new news window into the status page. At this moment it is just a test. 
The news inside the window are about peer arrivals and departures, remote search accesses and crawls

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4739 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-04-26 01:00:10 +00:00
orbiter
bac38cfa18 added very rudimentary peer news as rss feed. An example can be retrieved with 
http://localhost:8080/xml/feed.rss?channel=PEERNEWS
to be extended and integrated in interface ...

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4738 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-04-24 23:30:13 +00:00
orbiter
6495227ad6 the class rssReader is replaced by RSSReader, RSSFeed and RSSMessage 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4737 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-04-24 21:45:43 +00:00
orbiter
724bbdf9b2 refactoring of RSS reader 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4736 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-04-24 21:31:07 +00:00
orbiter
b9a2a2d287 more search performance hacks 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4735 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-04-24 15:09:06 +00:00
orbiter
ff755fb858 small corrections and enhancements after search timing profiling 
search should be a little bit faster now

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4734 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-04-24 13:31:55 +00:00
orbiter
0702dd2507 added a profiling script to analyse search process timing 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4733 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-04-24 13:28:18 +00:00
orbiter
d0e2830e01 enhanced the thread dump to make it usable for scripted remote-debugging 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4732 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-04-24 13:25:38 +00:00
orbiter
e024e3b9cf added new default profiles to distinguish snippet fetch for local and global search 
the difference is, that a local search will no not cause a re-indexing of loaded pages

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4731 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-04-24 08:42:08 +00:00
orbiter
2c0c8f0f0c SRU compliance according to 
http://www.loc.gov/standards/sru/specs/search-retrieve.html
The example given on this page can be used to retrieve opensearch-compatible rss pages with YaCy

Try it:
The transcription to YaCys search servlet address is
http://localhost:8080/yacysearch.rss?version=1.1&operation=searchRetrieve&query=dinosaur&maximumRecords=1&recordSchema=dc

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4730 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-04-23 16:16:41 +00:00
danielr
9b03310f8a bin jetzt wach :/ 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4729 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-04-23 07:50:21 +00:00
danielr
7bd8601f04 delete old releases compatible with java 1.5 ;) 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4728 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-04-23 07:22:20 +00:00
orbiter
e90282da1c added experimental javascript terminal for rss feeds (not used yet anywhere yet, expect the worst) 
.. possibly to be used as content for iframes within monitoring pages
not ready yet!

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4727 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-04-22 23:09:24 +00:00
danielr
da386a1924 fixed deleteOldDownloads if there are no downloads 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4726 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-04-22 21:36:52 +00:00
danielr
21418a22a3 removed DEBUG output 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4725 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-04-22 17:14:34 +00:00
danielr
79a3edeeef deleting downloaded releases after x days (default 30) 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4724 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-04-22 16:53:53 +00:00