Marown/yacy_search_server
1
0
Fork 0
mirror of https://github.com/yacy/yacy_search_server.git synced 2024-09-21 00:00:13 +02:00
Code Issues Actions Packages Projects Releases Wiki Activity
7,145 Commits 1 Branch 20 Tags 836 MiB
5892fff51f
Commit Graph

673 Commits

Author SHA1 Message Date
orbiter
5892fff51f introduction of dht-burst modes: this can expand the number of target peers in some cases where a better heuristic is needed. The problematic cases are either when a muti-word search is made (still a hard case for our term-oriented DHT) or when a network operator wants that all robinson peers are asked. We therefore introduced two new network steering values that switch on more peers during the peer selection. Because the number of peers can now be very large, the number of maximum httpc connections was also increased. 
Please see new coments in yacy.network.freeworld.unit for details of the new DHT selection methods.
The number of maximum peers is now not fixed to a specific number but may increase with
- the partition exponent
- the number of redundant peers
- the robinson burst percentage
- the multiword burst percentage
The maximum can then be the number of senior peers (all visible peers).

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7479 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2011-02-13 17:37:28 +00:00
orbiter
4588b5a291 - fixed document number limitation for crawls that restrict the number of documents per domain 
- some restructuring of the document counting and logging structures was necessary
- better abstraction of CrawlProfiles
- added deletion of logs to the index deletion option (if the index is deleted using the servlets) which is necessary to reset the domain counters for the page limitation
- more refactoring to get the LibraryProvider more clean
- some refactoring of the Condenser class

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7478 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2011-02-12 00:01:40 +00:00
orbiter
88773e4daa changed the default port from 8080 to 8090 
see also: http://forum.yacy-websuche.de/viewtopic.php?p=21683#p21683

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7454 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2011-01-28 10:54:13 +00:00
f1ori
4e29e9712a * create cleanupjob for cached failed urls 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7437 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2011-01-17 15:04:00 +00:00
orbiter
10ae8d961b - cora package has now no dependencies to other yacy packages and becomes a 'base' package (refactoring) 
- cleaned up (removed special code and documentation for 27c3)
- added remote search functions to be used within cora

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7420 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2011-01-03 20:52:54 +00:00
orbiter
db99db4be9 some redesign of the search-fail-response mechanism: 
when a search fails for a single url because the snippet cannot be generated, then the url reference is deleted from the index. This mechanism was redesign and enhanced. The process now also writes into the work tables into the table searchfl to prepare a re-indexing mechanism.

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7364 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2010-12-06 14:34:58 +00:00
orbiter
d4a1a1850b removed warnings 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7347 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2010-11-29 07:52:10 +00:00
low012
e7552bd719 *) cleaning up the code a little bit 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7343 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2010-11-27 00:54:59 +00:00
apfelmaennchen
737aaf6952 various small changes to ymarks 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7339 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2010-11-25 21:16:47 +00:00
apfelmaennchen
8a50670546 some code clean up for the last post 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7338 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2010-11-24 23:40:55 +00:00
apfelmaennchen
442497868d another step towards an auto tagging function for YMarks 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7337 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2010-11-24 23:26:29 +00:00
low012
eb79b952ef *) cleaner code 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7331 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2010-11-21 03:39:53 +00:00
low012
38fdf43587 *) renamed classes according to standard Java coding conventions 
*) String.isEmpty() was introduced in Java 1.6, but we still use Java 1.5

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7330 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2010-11-21 01:29:32 +00:00
low012
025e3f4790 *) renamed classes according to standard Java coding conventions 
*) removed unsused code

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7328 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2010-11-21 00:39:21 +00:00
low012
3b9aa0504e *) removed unsused code 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7327 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2010-11-21 00:28:32 +00:00
low012
db3db0fdb9 *) trying to make this class less confusing (probably failing) 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7326 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2010-11-21 00:13:08 +00:00
apfelmaennchen
54e63b556e intermediate step for a YMark auto-tagging function based on word frequencies. 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7325 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2010-11-17 15:17:29 +00:00
apfelmaennchen
403ee9c014 added a drill-down for metadata and word count to /api/ymarks/test_treeview.html 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7324 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2010-11-16 00:48:38 +00:00
apfelmaennchen
11ae5b108e enabled rebuildIndex for /Table_YMark_p.html (rebuilds the tags and folders index) 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7320 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2010-11-13 13:02:56 +00:00
apfelmaennchen
94a9be18a4 added a ymark table administration: /Table_YMark_p.html 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7316 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2010-11-10 22:53:27 +00:00
apfelmaennchen
25339f93c7 more updates to ymarks 
- working xbel import/export
- exported xbel includes yacy specific metadata but still validates against PUBLIC DTD


git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7315 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2010-11-09 17:01:31 +00:00
apfelmaennchen
cdd65aca71 update to ymarks 
- get_xbel.xml is almost working
- startet ymark api documentation info.html

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7313 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2010-11-07 20:03:01 +00:00
apfelmaennchen
808edffaf6 ymarks 
- some refactoring
- working xbel and html import (/api/ymarks/test_import.html)
- working treeview (/api/ymarks/test_treeview.html)

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7312 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2010-11-06 20:26:13 +00:00
orbiter
becc463d8a enhanced did-you-mean 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7300 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2010-11-04 00:25:19 +00:00
apfelmaennchen
43586a2ace a update to ymarks (please test if you wish): 
- import HTML (e.g. FF export) via /api/ymarks/import.html
- view your import via /api/ymarks/test.html
- get a xml list via /api/ymarks/get_ymark_list.xml?tags=&folders=
- delete bookmark tables via standard interface /Tables_p.html
it is still very experimental!! 

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7299 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2010-11-03 22:52:03 +00:00
apfelmaennchen
f5324b27f2 more updates to the new bookmarks (ymarks).... 
- split YMarkTables and YMarkIndex in two different classes
- HTML import is working properly
- XBEL import is still broken


git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7292 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2010-11-03 06:47:02 +00:00
orbiter
70c95608d4 Added CORS Access header for yacysearch.rss output 
used some of the recommendations from Copro:
http://forum.yacy-websuche.de/viewtopic.php?p=21015#p21015
Original Request:
http://forum.yacy-websuche.de/viewtopic.php?p=20829#p20829

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7288 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2010-11-02 16:28:40 +00:00
apfelmaennchen
efe0667fdd more new bookmark (ymarks) code with experimental html and xbel import 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7281 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2010-10-28 15:24:15 +00:00
f1ori
7d8de34778 * add a bit documentation to DigestURI, use DigestURI(string) instead of DigestURI(string, null) 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7276 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2010-10-26 16:10:20 +00:00
apfelmaennchen
d0e6c03b51 some updates to the new bookmark code... 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7272 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2010-10-25 22:44:05 +00:00
low012
65a0381f76 *) cleaning up code (still not done) 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7267 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2010-10-24 23:57:07 +00:00
apfelmaennchen
9c94ebdee4 small changes to new bookmark code... 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7265 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2010-10-22 13:14:09 +00:00
apfelmaennchen
244b56e9d3 an update to the new bookmark code... 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7264 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2010-10-21 19:18:17 +00:00
low012
dc40f51b8d *) added headlines as proposed by Vega 
*) <pre> will be displayed monospaced in wiki and blog again
*) bugfix for <pre> spanning multiple lines
*) replaced deprecated <s> tag with <span> equivalent

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7262 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2010-10-18 23:00:43 +00:00
apfelmaennchen
f035f257da added some more bookmark code... 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7261 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2010-10-18 21:09:41 +00:00
low012
22ed9c380c *) fixed bug which was introduced in r7226 (shame on me) which made wiki unusable (all entries were stored with empty subject as key -> edits were lost) 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7260 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2010-10-18 21:06:23 +00:00
orbiter
58e74282af added a word counter statistic in condenser which is used by the did-you-mean to calculate best matches for given search words. 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7258 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2010-10-18 11:35:09 +00:00
apfelmaennchen
a79728b97d some updates to experimental bookmark code... 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7254 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2010-10-17 09:58:50 +00:00
apfelmaennchen
ef782cd026 and even more experimental bookmark code... 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7253 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2010-10-16 10:20:41 +00:00
orbiter
ca738ac924 - added a tag cloud to search results (using the topics) 
- some refactoring of score classes
- added default package for new classes add_ymark and delete_ymark

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7251 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2010-10-15 22:01:39 +00:00
apfelmaennchen
7aca763ca8 Some more experimental bookmark code... 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7250 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2010-10-15 12:53:41 +00:00
apfelmaennchen
4270ed696c Experimental code (I need to transfer the code to my macbook, sorry) for the new bookmarks API based on the Tables concept (same as for crawl starts). Currently you can add a bookmark by api/ymarks/add_ymark.xml?url=http://www.yacy.net&title=YaCy and watch the result via the standard view Tables_p.html. 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7249 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2010-10-15 05:40:19 +00:00
orbiter
e4d561971e added more score cluster options and made score cluster usage more transparent 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7248 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2010-10-14 11:40:02 +00:00
orbiter
7cd9d9d22a - enhanced DidYouMean computation using a faster count on index entries; this causes that results can be ranked better 
- added limitations on DidYouMean result sets according to input and output string length

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7246 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2010-10-12 22:02:10 +00:00
orbiter
de722090b5 enhancements in did-you-mean guessing 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7243 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2010-10-12 09:45:15 +00:00
orbiter
a59c885ee0 autocomplete and did-you-mean can now understand _all_ languages and can generate suggestions in all languages and character types 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7242 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2010-10-12 08:36:33 +00:00
low012
b9f405d1e8 *) added comments 
*) more beautyful and easier to understand code (IMO)
*) added display= parameter to a lot of links in Wiki.html

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7226 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2010-10-08 00:32:50 +00:00
orbiter
e63896f2a8 added an intranet scanner and a servlet which shows all intranet addresses and an option to start a site-crawl for all these addresses at once. 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7203 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2010-09-28 12:18:54 +00:00
low012
afa708d552 *) added <s>...</s> tag to WikiCode -> works just as the HTML equivalent 
*) code changes (PMD) without functional changes

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7193 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2010-09-26 12:57:07 +00:00
low012
5a9ea0308f *) further simplification of wiki code parser (less redundancy in code, less magic numbers), still not done with it... 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7166 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2010-09-18 11:40:37 +00:00