Marown/yacy_search_server
1
0
Fork 0
mirror of https://github.com/yacy/yacy_search_server.git synced 2024-09-21 00:00:13 +02:00
Code Issues Actions Packages Projects Releases Wiki Activity
5,513 Commits 1 Branch 20 Tags 836 MiB
635b0a9da7
Commit Graph

5513 Commits

This Branch
This Branch
All Branches
Author SHA1 Message Date
lotus
635b0a9da7 code-split 
allow cgi indexing

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5839 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-04-20 13:28:28 +00:00
orbiter
e7559f3234 fix for http://forum.yacy-websuche.de/viewtopic.php?p=13977#p13977 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5838 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-04-20 10:06:55 +00:00
orbiter
fa3adbbfc6 added domain checks to surrogate reader and RWI transfer receiver to prevent spaming using surrogates 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5837 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-04-20 06:38:28 +00:00
f1ori
76af84d732 * add custom comparator to ScoreCluster for byte[] 
* fixes http://forum.yacy-websuche.de/viewtopic.php?f=6&t=2010


git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5836 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-04-19 20:01:46 +00:00
low012
31c6934df2 *) fix for r5832 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5835 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-04-19 07:40:23 +00:00
lotus
ab0030d7a7 allow dht-out for remote-crawl processing peers on default settings 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5834 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-04-18 20:04:01 +00:00
lotus
616a4d724f high-end favicon with 2 versions: 
* true color + alpha channel for modern browsers
* 256 colors and non-transparent background for others

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5833 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-04-18 18:37:26 +00:00
low012
d1116c049f *) added new method "contains()" to Blacklist interface 
*) implemented contains() in class AbstractBlacklist
*) used new method in Blacklist_p to prevent double entries in blacklists

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5832 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-04-18 16:27:17 +00:00
f1ori
08445e42f0 * don't throw exception, in case of bad charset in http-header 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5831 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-04-18 15:38:29 +00:00
f1ori
2f860a2564 * convert byte[] hashes to string for log output 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5830 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-04-18 14:35:18 +00:00
f1ori
94a6c83256 * rewrite code without using java 1.6 features 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5829 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-04-17 15:54:44 +00:00
f1ori
d93a2a6552 * ignore whitespaces so you can copy&paste signatures better 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5828 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-04-17 14:52:42 +00:00
lulabad
fadf311b97 added sign key for yacystats updates 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5827 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-04-17 14:32:08 +00:00
orbiter
fbcbcc5bdb export of yacy document objects as dublin core record in xml 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5826 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-04-17 14:20:12 +00:00
orbiter
d7cbf4cdd4 more performance hacks: less overhead in word hash computation 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5825 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-04-17 13:47:06 +00:00
orbiter
29e96c1a60 bugfixes and performance hacks 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5824 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-04-17 13:04:56 +00:00
orbiter
4e97a31009 corrections in dublin core syntax 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5823 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-04-17 12:23:00 +00:00
f1ori
44daec7936 * introduce signatures to autoupdate 
as long as there aren't publickeys for the updatelocations set,
  no signatures are checked
* wiki-article follows...


git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5822 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-04-17 09:58:06 +00:00
orbiter
538e375901 replaced old caching method for computed word hashes with a better method. The word hash computation is a new performance bottleneck (after the IO bottleneck was removed with the IndexCell data structure) and a better caching for word hashes was necessary. 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5821 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-04-17 09:26:16 +00:00
orbiter
9e853e1977 partly reverting SVN 5818: identical comparator required for join operator 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5820 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-04-17 08:18:01 +00:00
orbiter
e16c25ddf7 (peak-) performance hacks 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5819 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-04-16 22:45:39 +00:00
orbiter
63cd152969 fixes 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5818 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-04-16 22:18:35 +00:00
orbiter
7dfe7e7cc6 fixed some problems with surrogate reader. This is now ready for testing. 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5817 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-04-16 21:29:41 +00:00
orbiter
3a1364ed5c removed example lines from SurrogateReader sources; added additional example file 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5816 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-04-16 21:05:34 +00:00
orbiter
9050a3c4c5 alpha version of surrogate reading and indexing. 
see the example file for an explanation.

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5815 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-04-16 20:47:55 +00:00
orbiter
870066ab35 another fix 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5814 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-04-16 20:23:20 +00:00
orbiter
b15b059c0d fix for latest commit 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5813 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-04-16 19:53:21 +00:00
orbiter
c8624903c6 full redesign of index access data model: 
terms (words) are not any more retrieved by their word hash string, but by a byte[] containing the word hash.
this has strong advantages when RWIs are sorted in the ReferenceContainer Cache and compared with the sun.java TreeMap method, which needed getBytes() and new String() transformations before.
Many thousands of such conversions are now omitted every second, which increases the indexing speed by a factor of two.

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5812 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-04-16 15:29:00 +00:00
f1ori
dd6b5005ff * fix missing charset handling in getpageinfo_p 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5811 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-04-16 12:31:28 +00:00
orbiter
bd5f4c78d8 - added default profile for surrogate indexing 
- integrated surrogate indexing into indexing queue process

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5810 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-04-16 08:01:38 +00:00
orbiter
ad78e3a59f - less lines in rssTerminal 
- crawl more documents: if remote crawling is enabled, a remote crawl list is also loaded if a local crawl is running in case that the indexer is idle

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5809 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-04-15 23:07:51 +00:00
orbiter
bc80dc913a added new surrogate reader (surrogates are parsed documents on batches) 
this will open a new way to insert indexes to YaCy (instead crawling)

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5808 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-04-15 15:30:25 +00:00
orbiter
12d81e98eb - fixed bad search results when searching for empty string 
- simplified result handling and page composition in case that nothing was searched


git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5807 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-04-15 11:22:43 +00:00
orbiter
8a24350036 - fix for join method with new generalized RWI data structure (caused by latest commit) 
- added more functions to mediawiki parser


git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5806 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-04-15 10:26:24 +00:00
orbiter
e58320a507 added more info in log fore debugging 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5805 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-04-15 07:37:36 +00:00
orbiter
89ec3acb3e - full abstraction of index content type: the kelondro full text index may now also contain indexes about other content than text, i.e. navigation indexes or reverse linking indexes. 
- during index joins all word positions are maintained: better ranking for word distance possible; exact phrase match can be implemented soundly


git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5804 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-04-15 06:34:27 +00:00
borg-0300
7a48090fcf - fix for "uk" language 
- svn attributes added

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5803 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-04-14 11:40:44 +00:00
orbiter
dc2af61bc9 allow up to 50 results from remote peers 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5802 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-04-13 21:47:57 +00:00
orbiter
c0e8ed5461 fixed problem with not http client 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5801 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-04-13 21:21:47 +00:00
low012
6504b21cea *) fix for http://forum.yacy-websuche.de/viewtopic.php?t=1976 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5800 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-04-13 09:22:11 +00:00
orbiter
8862a2fed0 ups 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5799 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-04-12 10:22:21 +00:00
orbiter
de68948bc5 better handling of free memory computation and emrgency cache flush for index cell 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5798 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-04-12 09:24:32 +00:00
orbiter
601d63ef48 removed comment tag (no use at this point) 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5797 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-04-11 12:24:45 +00:00
low012
c2d85b039e *) added language statistics files 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5796 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-04-11 10:18:42 +00:00
low012
0c8fd811dc *) first and very limited version of XML import, does not use benefits provided by XML yet 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5795 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-04-11 09:55:46 +00:00
f1ori
fcb77c3140 * added .im (Isle of Man) to TLD-list 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5794 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-04-10 23:06:48 +00:00
orbiter
8ce5bb4f31 added shell scripts that list host addresses 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5793 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-04-10 09:45:22 +00:00
orbiter
51ea865569 small fix for localsearch shell script 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5792 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-04-10 09:44:03 +00:00
orbiter
b81c7467d8 protection against too many files in RICELL in case of massive emergency dumps caused by low memory 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5791 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-04-09 23:55:47 +00:00
orbiter
d4d87d90c4 - extended experimental wikipedia dump parser 
- removed historic, possibly unused code from wiki parser that was in conflict with actual wikipedia wiki code

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5790 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-04-09 14:55:20 +00:00