Marown/yacy_search_server
1
0
Fork 0
mirror of https://github.com/yacy/yacy_search_server.git synced 2024-09-21 00:00:13 +02:00
Code Issues Actions Packages Projects Releases Wiki Activity
7,501 Commits 1 Branch 20 Tags 836 MiB
11dc653de3
Commit Graph

5013 Commits

Author SHA1 Message Date
orbiter
11dc653de3 added a visualization of peer pings to the performance graphic 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7837 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2011-07-14 07:07:06 +00:00
orbiter
3a191cdf14 because newbies are scared about the memory consumption in the performance graph and arguments about high memory consumption according to bad knowledge about java garbage collection techniques, the memory display had been removed from the performance graph shown on the Status.html page. The memory graph can still be seen on the Performance page where the memory graph is just like it was. 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7836 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2011-07-14 03:25:57 +00:00
cominch
09bb7a390c do not replace malformed or invalid URLs in urlproxy 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7835 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2011-07-12 07:44:23 +00:00
orbiter
52d799e7c8 fix for solr auth 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7833 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2011-07-05 09:21:30 +00:00
orbiter
9eb8e9acd9 no error message about missing browser in headless environments 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7832 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2011-07-05 06:54:05 +00:00
orbiter
d3c89b90ce temporary adding the old httpclient-3.1 again because the solrj classes need them. should be removed as soon solrj supports httpclient-4 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7831 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2011-07-04 17:04:49 +00:00
orbiter
bd99969758 fixed bad query 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7830 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2011-07-04 16:53:18 +00:00
orbiter
768c59740c - replaced solrj 3.1 with solrj 3.3 
- updated also slf4j
- added authentication for solrj


git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7829 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2011-07-04 16:35:30 +00:00
low012
c7b95e8c81 *) Invalid crawl profiles (containing invalid mustmatch/mustnotmatch filters) will be moved from active crawls to invalid crawls (new file: DATA/INDEX/freeworld/QUEUES/crawlProfilesInvalid.heap). This file can not be edited yet, but it shoudl be easy to extend the CrawlProfileEditor accordingly. 
*) Corrupt crawlProfilesPassive.heap would cause crawlProfilesActive.heap to be deleted. Don't know if this ever happend, but will not happen anymore.
*) Cleaned up a little bit.
*) Added some comments.

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7827 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2011-07-03 23:55:55 +00:00
orbiter
6d2e252bcf fix for: 
java.lang.NullPointerException
	at net.yacy.kelondro.index.RowCollection.<init>(RowCollection.java:97)
	at net.yacy.kelondro.index.RowSet.<init>(RowSet.java:48)
	at net.yacy.kelondro.rwi.ReferenceContainer.<init>(ReferenceContainer.java:58)
	at net.yacy.kelondro.rwi.ReferenceIterator.next(ReferenceIterator.java:69)
	at net.yacy.kelondro.rwi.ReferenceIterator.next(ReferenceIterator.java:43)
	at net.yacy.kelondro.blob.ArrayStack.merge(ArrayStack.java:1023)
	at net.yacy.kelondro.blob.ArrayStack.mergeWorker(ArrayStack.java:922)
	at net.yacy.kelondro.blob.ArrayStack.mergeMount(ArrayStack.java:869)
	at net.yacy.kelondro.rwi.IODispatcher$MergeJob.merge(IODispatcher.java:267)
	at net.yacy.kelondro.rwi.IODispatcher$MergeJob.access$300(IODispatcher.java:239)
	at net.yacy.kelondro.rwi.IODispatcher.run(IODispatcher.java:180)

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7822 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2011-07-03 20:44:33 +00:00
orbiter
719777b2a7 replaced method to call getUsableSpace using reflection with direct call since we now use java 1.6 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7821 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2011-07-03 18:13:37 +00:00
orbiter
2d4bb139d3 - added counting of links with noindex tag for solr index 
- bugfixes for solr index

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7820 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2011-07-03 06:40:05 +00:00
orbiter
892caccdca added default configuration in ConfigurationSet in case of new values 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7814 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2011-07-02 00:09:49 +00:00
orbiter
bda3eec0ff added parsing of canonical link element to html parser 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7812 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2011-07-01 16:38:01 +00:00
orbiter
b6f09a475d - added an index profile editor in the /indexFederated_p.html servlet for solr indexes 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7811 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2011-06-30 15:49:21 +00:00
orbiter
b666a929e7 fixed Semaphore handling in case of interruptions 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7809 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2011-06-30 15:37:14 +00:00
orbiter
de7a054d77 added parser for such files like the new solr.key.list 
it parses text files with the following syntax:
 - all lines beginning with '##' are comments
 - all non-empty lines not beginning with '#' are keyword lines
 - all lines beginning with '#' and where the second character is not '#' are commented-out keyword lines


git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7808 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2011-06-29 15:35:45 +00:00
f1ori
a17351dcfe * navigation bar for filetype constraints 
javascript interpreted backslashes from urlmask as escaping and didn't forward them to yacy

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7806 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2011-06-29 15:30:24 +00:00
f1ori
96957375cc * fix url proxy for relative links and chromium 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7805 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2011-06-29 09:32:02 +00:00
orbiter
9ebc75db4b fix for channel authorization 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7803 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2011-06-26 23:14:02 +00:00
orbiter
267290a821 removed the semaphores from the cache dump process because I believe some of the semaphores may be lost somewhere which then causes that the cache is never flushed and then the peer dies from a OOM. The re-introduced synchronization may not be the best solution but should ensure that the caches are flushed. 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7802 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2011-06-26 21:45:04 +00:00
orbiter
6d9e5865ee faster appearance of search result page (but complete search time is the same) 
this was inspired by http://bugs.yacy.net/view.php?id=37

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7801 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2011-06-26 21:17:02 +00:00
orbiter
f7ca84cfc0 enhanced template engine 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7800 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2011-06-26 21:15:13 +00:00
orbiter
d8072d1866 added more info to DNS cache in /PerformanceMemory_p.html 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7798 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2011-06-24 08:27:36 +00:00
orbiter
f803da8aae code cleanup 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7797 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2011-06-24 00:24:00 +00:00
orbiter
84c9658644 added a file type navigator 
added a protocol navigator

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7795 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2011-06-23 15:39:52 +00:00
orbiter
31283ecd07 - added a search option to filter only specific network protocols. i.e. get only results from ftp servers. Just add '/ftp' to your search. 
for example search for "passwd /ftp". This can also be done with /http /https and /smb
- fixed some search throttling processes that should protect your peer against search DoS or strong search load

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7794 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2011-06-23 11:57:17 +00:00
orbiter
4b425ffdd2 fix for http://bugs.yacy.net/view.php?id=41 
added another RSS channel "PROXY". the rss feed for peer news filters this channel if there is not an authorized access on that channel


git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7792 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2011-06-22 10:19:32 +00:00
orbiter
7db208c992 performance hacks: more pre-allocated StringBuilder 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7790 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2011-06-21 23:10:50 +00:00
orbiter
87bd559c42 fixed warning 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7789 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2011-06-20 22:53:43 +00:00
orbiter
07e89a7ae5 added @Deprecated 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7788 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2011-06-20 22:33:45 +00:00
orbiter
9706fc55aa enhanced content scraper (should discover urls much faster in case of very large plain texts) 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7787 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2011-06-20 22:29:45 +00:00
orbiter
996f0a8764 disabled assert in Base64Order which eats away too much performance during testing with -l 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7786 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2011-06-19 13:34:55 +00:00
orbiter
f667b9c289 enhanced identificator: using AtomicInteger for counter 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7785 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2011-06-19 13:31:10 +00:00
orbiter
16327d1cbe unwrapping of call depth (one call less for UTF8.String) 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7784 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2011-06-19 13:15:01 +00:00
orbiter
f30d36b101 enhanced template engine 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7783 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2011-06-19 13:02:06 +00:00
orbiter
aa6c32d753 enhanced UTCDiffString 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7782 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2011-06-19 12:38:06 +00:00
f1ori
f87865a50b always shutdown log, fixes zombie processes in init stop script 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7780 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2011-06-15 09:14:51 +00:00
orbiter
115abc8917 - more attributes for search progress bar 
- moved cache strategy to cora package

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7778 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2011-06-13 21:44:03 +00:00
sixcooler
7bfa6bb4b6 prevent getting a yacySeed from zero-length-hash-string by chance 
(for eg.: proxy-crawls got displayed as initiated by some other peer)

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7776 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2011-06-05 22:58:17 +00:00
orbiter
bce280a308 update on options for interface graphics 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7775 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2011-06-05 22:48:21 +00:00
orbiter
77fe69395d added jempbox-1.5.0.jar which is required by pdfbox-1.5 as stated in http://pdfbox.apache.org/dependencies.html 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7774 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2011-06-05 20:04:41 +00:00
sixcooler
df1725ef43 re-enable POST over proxy, which didn't work since update to httpcore-4.1.1 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7772 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2011-06-04 13:25:03 +00:00
orbiter
2683162ec5 - added more options to access grid picture, web structure picture and network graphics 
- remove test class


git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7770 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2011-06-02 23:27:26 +00:00
orbiter
0c1b29f3c9 - applied many small performance hacks 
- added a memory limitation in the zip parser and the pdf parser
- added a search throttling: if there are too many search queries are still to be computed, then new requests are not accepted for some time. if after a one second still no space is there to perform another search, the search terminates with no results. this case should only happen in case of DoS-like situations and in case of strong load on a peer like if it is integrated in metager.
- added a search cache deletion process that removes search requests in case that throttling happens

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7766 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2011-06-01 19:31:56 +00:00
f1ori
900dacbf97 * improve link rewriting in proxy-url 
* only rewrites links, which are in current search domain

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7765 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2011-06-01 13:27:04 +00:00
f1ori
dc855d881b * further improve proxyurl 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7762 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2011-05-30 21:25:20 +00:00
orbiter
a7a6b392f5 code cleanup 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7760 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2011-05-30 10:16:43 +00:00
orbiter
fe0c08455b more concurrency (enhancement) hacks 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7759 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2011-05-30 08:53:58 +00:00
orbiter
0e9a99cb05 another resource hack 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7758 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2011-05-30 07:51:18 +00:00