Marown/yacy_search_server
1
0
Fork 0
mirror of https://github.com/yacy/yacy_search_server.git synced 2024-09-19 00:01:41 +02:00
Code Issues Actions Packages Projects Releases Wiki Activity
7,646 Commits 1 Branch 20 Tags 836 MiB
813f297a95
Commit Graph

7646 Commits

This Branch
This Branch
All Branches
Author SHA1 Message Date
lotus
07cbb6cb5f display cache hit/miss values in correct column 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7781 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2011-06-15 18:57:04 +00:00
f1ori
f87865a50b always shutdown log, fixes zombie processes in init stop script 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7780 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2011-06-15 09:14:51 +00:00
orbiter
115abc8917 - more attributes for search progress bar 
- moved cache strategy to cora package

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7778 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2011-06-13 21:44:03 +00:00
suessthomas
ccad615f58 The Java-XMS and Xmx values for the target of "run" (run YaCy) inserted. 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7777 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2011-06-11 21:22:08 +00:00
sixcooler
7bfa6bb4b6 prevent getting a yacySeed from zero-length-hash-string by chance 
(for eg.: proxy-crawls got displayed as initiated by some other peer)

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7776 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2011-06-05 22:58:17 +00:00
orbiter
bce280a308 update on options for interface graphics 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7775 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2011-06-05 22:48:21 +00:00
orbiter
77fe69395d added jempbox-1.5.0.jar which is required by pdfbox-1.5 as stated in http://pdfbox.apache.org/dependencies.html 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7774 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2011-06-05 20:04:41 +00:00
lotus
72a3cd5832 equalize lock icon for Status.html 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7773 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2011-06-04 18:55:09 +00:00
sixcooler
df1725ef43 re-enable POST over proxy, which didn't work since update to httpcore-4.1.1 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7772 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2011-06-04 13:25:03 +00:00
suessthomas
66c477129e Creates a new network definition, yacy.networks.metager.unit. 
The YaCy freeworld network used in this network definition, minor enhancements for the feed of MetaGer were integrated.

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7771 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2011-06-03 22:34:42 +00:00
orbiter
2683162ec5 - added more options to access grid picture, web structure picture and network graphics 
- remove test class


git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7770 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2011-06-02 23:27:26 +00:00
sixcooler
efcd21e0ed new httpclient, httcore (bugfixrelease) 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7769 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2011-06-02 21:34:50 +00:00
orbiter
d0d6123b18 added a deploy script that can be used to deploy yacy releases into the current release for testing 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7768 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2011-06-01 19:52:05 +00:00
orbiter
265b7ce4f9 removed pause in search test 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7767 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2011-06-01 19:49:44 +00:00
orbiter
0c1b29f3c9 - applied many small performance hacks 
- added a memory limitation in the zip parser and the pdf parser
- added a search throttling: if there are too many search queries are still to be computed, then new requests are not accepted for some time. if after a one second still no space is there to perform another search, the search terminates with no results. this case should only happen in case of DoS-like situations and in case of strong load on a peer like if it is integrated in metager.
- added a search cache deletion process that removes search requests in case that throttling happens

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7766 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2011-06-01 19:31:56 +00:00
f1ori
900dacbf97 * improve link rewriting in proxy-url 
* only rewrites links, which are in current search domain

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7765 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2011-06-01 13:27:04 +00:00
sixcooler
7fea51ecee check filter to bee a correct pattern on edit CrawlProfiles 
see; http://forum.yacy-websuche.de/viewtopic.php?f=5&t=3277&p=22662#p22660

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7764 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2011-05-31 16:13:33 +00:00
orbiter
af63aa1d0e added fresh links to java regular expression api-doc 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7763 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2011-05-31 13:33:04 +00:00
f1ori
dc855d881b * further improve proxyurl 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7762 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2011-05-30 21:25:20 +00:00
orbiter
761b1c71dc added latest pdfbox 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7761 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2011-05-30 14:56:36 +00:00
orbiter
a7a6b392f5 code cleanup 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7760 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2011-05-30 10:16:43 +00:00
orbiter
fe0c08455b more concurrency (enhancement) hacks 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7759 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2011-05-30 08:53:58 +00:00
orbiter
0e9a99cb05 another resource hack 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7758 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2011-05-30 07:51:18 +00:00
orbiter
535b6b953c more hacks to omit superfluous string object allocation 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7757 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2011-05-30 07:31:17 +00:00
orbiter
87082f407e less String object creation during search 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7756 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2011-05-30 04:19:20 +00:00
orbiter
ab5a16b957 lesse memory occupation during ranking and faster host navigator 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7755 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2011-05-29 20:33:12 +00:00
lotus
b8aa41a1b4 show nsis version in installer again for more transparency 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7754 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2011-05-27 16:30:55 +00:00
orbiter
1489ebeedf one more hack to free ram for search events 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7753 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2011-05-27 14:26:37 +00:00
orbiter
3c2b994bd6 write access/load time to solr index 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7752 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2011-05-27 12:35:08 +00:00
orbiter
a36fda991e hack to increase speed of url hash computation 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7751 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2011-05-27 12:34:38 +00:00
orbiter
752576b521 - localsearch test script does also a snippet-fetch 
- killYACY.sh does not need a sleep between kill -3 and kill -9

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7750 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2011-05-27 12:08:45 +00:00
f1ori
ddcc333acc * fix negative result counts 
results sorted out by add to RankingProcess were counted in
sortedout-counter, but were not added to remote_indexCount nor
local_indexCount

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7749 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2011-05-27 11:21:00 +00:00
orbiter
fa734bdf9f better memory protection in search logger 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7748 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2011-05-27 11:18:22 +00:00
orbiter
dbea40d536 - changed snippet fetch strategy logic: do not check if entry is in cache. This should reduce IO load on the HTCACHE which is a showstopper during large number of search requests 
- forced a possible short memory status when a search is started to flush caches that may cause search-heaps with resource contention effects

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7747 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2011-05-27 09:32:03 +00:00
orbiter
4bea3f9714 hack to reduce resource contention caused by massive UTF8 decodings which use java.nio resources: 
used a ASCII String <-> byte[] conversion wherever possible. Many Strings in YaCy are hashes which are pure ASCII (base64 hashes).
The new ASCII String <-> byte[] conversion method have less computation overhead than the UTF8 conversion.

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7746 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2011-05-27 08:24:54 +00:00
orbiter
746e3c3b06 Replaced a widely-used Property Object in the httpd with HashMap<String, Object> which is not synchronized like Properties 
A synchronization is not needed here and applies an overhead to the httpd process which is now removed.

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7745 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2011-05-26 16:34:35 +00:00
orbiter
cc239b18cd fix for IPv6 localhost proxy client 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7744 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2011-05-26 16:24:11 +00:00
f1ori
fcb7525047 * add .gitignore to svn, so it doesn't get lost on git syncronization 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7743 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2011-05-26 16:13:07 +00:00
f1ori
14e1666b21 * fix replacing regexes in url proxy 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7742 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2011-05-26 16:09:29 +00:00
orbiter
e28bd0d038 fix for some possible causes of memory leaks 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7741 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2011-05-26 14:35:32 +00:00
orbiter
09ba6814c0 - non-blocking word hash computation with dynamic digest object generation (this was important!) 
- (very) small performance enhancement in did-you-mean


git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7740 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2011-05-26 12:58:11 +00:00
orbiter
8d9b5dda3b disabled did-you-mean computation for json and rss search results where this info is not used 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7739 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2011-05-26 12:35:24 +00:00
orbiter
10e2f588f8 - enhanced ybr ranking computation 
- many speed/performance hacks
- added solr charding and new charding web interface
- added option to switch off the yacy index when using solr
- added new fail-url categories which are used to make a distinction which fail-urls to be sent to solr
- refactoring/renaming of some method names to distinguish host/url hashes better
- a large number of bug/npe fixes

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7738 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2011-05-26 10:57:02 +00:00
orbiter
bd55dcee50 - commented out experimental distributed ranking loading 
- less threads for blocking threads
- disable all threads for DHT transmission for networks with zero peers

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7737 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2011-05-24 21:08:01 +00:00
orbiter
98c4d25185 fix for endless loop in FTP crawling, see http://bugs.yacy.net/view.php?id=32 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7736 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2011-05-24 10:06:20 +00:00
orbiter
d1dbbd956a always use a template method cache even if the template cache flag is set to false. This flag is only used to make dynamic updates to the template files, to not dynamic updates to the rewrite methods (which is not possible without recompiling). low memory usage is guaranteed by the usage of soft references which are dropped before an OOM is thrown 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7735 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2011-05-24 09:31:07 +00:00
orbiter
0d040ff6bb fix for bug 0000036: no crawling of https pages 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7734 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2011-05-24 09:14:32 +00:00
orbiter
3ed4a09368 small features, some bug fixes and performance hacks 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7733 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2011-05-23 21:08:04 +00:00
orbiter
e55c254f7b enhanced logging 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7732 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2011-05-22 20:12:13 +00:00
orbiter
3ec94d87c4 show dom counter only for active crawls where the dom counter is enabled within the crawl profile 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7731 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2011-05-22 19:34:20 +00:00