Marown/yacy_search_server
1
0
Fork 0
mirror of https://github.com/yacy/yacy_search_server.git synced 2024-09-21 00:00:13 +02:00
Code Issues Actions Packages Projects Releases Wiki Activity
12,795 Commits 1 Branch 20 Tags 836 MiB
7717a3d43d
Commit Graph

68 Commits

Author SHA1 Message Date
Michael Peter Christen
9bf0d7ecb9 added a new collection type 'dht' to all documents from the peer-to-peer 
interface to distinguish rich and poor document data.
This also reverts some changes from commit
796770e070 because the firstSeen database
is the wrong method to distinguish these types of data
 2015-03-24 12:32:39 +01:00
Michael Peter Christen
3cd7deb3b8 do not flush non-errors to stdout because this is a concurrency issue. 
the flush-call appeared very often in thread dumps with high load, so
this hopefully gives some performances
 2014-12-28 15:48:37 +01:00
Michael Peter Christen
1a764135be one more Thread Dump fix for new bootstrap css style 2014-03-27 23:01:28 +01:00
Michael Peter Christen
bb21d825f9 fix for thread dump line spacing 2014-03-27 22:13:37 +01:00
orbiter
3c3cb78555 - removed a lot of garbage and bloated code from GuiHandler. 
- transformed log lines to String before they are stored because the
storage space is about 1:250 (45kb for one line before transformation,
180 bytes afterwards)
- this saves up to 10MB RAM so we can increase the number of lines to
1000 again.
 2013-10-24 20:42:34 +02:00
Michael Peter Christen
5afa6e3aee Automatically flush the log cache if a short memory status is reached. 
For the default of 200 lines this can flush about 10MB.
 2013-10-24 17:39:50 +02:00
Michael Peter Christen
6aabc4e5c8 reduced logging line memory, 10000 lines had filled up 450MB! grrr. 
(thank you, a bomb from the past)
 2013-10-24 16:17:53 +02:00
Michael Peter Christen
336f86394c replaced StringBuffer with StringBuilder 2013-07-23 12:21:27 +02:00
Roland Haeder
841a28ae76 Added 'final' for all exception blocks as this helps the Java compiler 
to optimize memory usage

Conflicts:
	source/net/yacy/search/Switchboard.java
 2013-07-17 18:31:30 +02:00
Michael Peter Christen
5c6946dd5f replaced usage of log4j by ConcurrentLog where possible 2013-07-09 14:42:39 +02:00
Michael Peter Christen
5878c1d599 - refactoring of log to ConcurrentLog: 
jdk-based logger tend to block
at java.util.logging.Logger.log(Logger.java:476) in concurrent
environments. This makes logging a main performance issue. To overcome
this problem, this is a add-on to jdk logging to put log entries on a
concurrent message queue and log the messages one by one using a
separate process.
- FTPClient uses the concurrent logging instead of the log4j logger
 2013-07-09 14:28:25 +02:00
Michael Peter Christen
35fa718b77 testing to use solr for portalsearch caused some bugfixing but no full 
success: try to comment out the solr search request in
yacy-portalsearch.js
 2013-02-25 14:31:50 +01:00
Michael Peter Christen
1baf498d59 - show more lines in online log 
- reverse order is default now
 2012-10-25 18:38:39 +02:00
Michael Peter Christen
2f536cb54d code cleanup: removed unised methods and made more methods and objects 
private
 2012-10-08 10:50:24 +02:00
Michael Peter Christen
00c1c777fa refactoring 2012-09-21 15:48:16 +02:00
Michael Peter Christen
f0a079ac9f allow larger log entries 2012-07-14 16:28:14 +02:00
Michael Peter Christen
f78ce93a80 collection of speed and memory saving hacks 2012-07-13 21:15:38 +02:00
orbiter
a196f24f60 prevent enqueueing of non-loggeable logging entries 2012-07-12 19:42:42 +02:00
orbiter
482afed07c reduced logging overhead (a bit) 2012-07-12 19:23:40 +02:00
Michael Peter Christen
83da68c4c1 fixed a memory leak inside the logger which appeared if the log was 
writter faster that the logger is able to print this out to its out
stream. A very large collection of unwritten log outputs had been seen
during strong crawling. The new ArrayBlockingQueue is limited to prevent
this case.
 2012-07-12 01:23:04 +02:00
orbiter
0cbda0b2b8 - replaced all length() == 0 and size() == 0 with isEmpty() 
- replaced some length() > 0 and size() > 0 with !isEmpty() - cannot be
done automatically
- implemented some isEmpty() methods
 2012-07-10 22:59:03 +02:00
Michael Peter Christen
f32de94723 more logging 2012-07-08 22:04:36 +02:00
Michael Peter Christen
b0c408788b made class methods static where possible 2012-07-05 12:38:41 +02:00
Michael Peter Christen
e0d8643226 - performance hacks 
- added log warnings in case that search processes run into time-out
situations
- better concurrency for Integer formatter (used a non-synchronized
formatter before)
- bugfix for search termination (a poison pill was missing)
- added timeout parameters for search (again) -> target is, that they
are never reached.
 2012-06-04 15:37:39 +02:00
Michael Peter Christen
8d997d55b6 better logging 2012-05-30 15:47:35 +02:00
Michael Peter Christen
43c2c6e588 better logging 2012-05-30 15:27:45 +02:00
Michael Peter Christen
3e1bc9477f Merge branch 'master' of ssh://git@gitorious.org/yacy/rc1.git 2012-05-17 13:58:09 +02:00
Roland 'Quix0r' Haeder
d10627d591 More sync in close() methods 
Conflicts:
	source/net/yacy/kelondro/logging/GuiHandler.java
	source/net/yacy/kelondro/workflow/InstantBusyThread.java
 2012-05-17 06:03:18 +02:00
Michael Peter Christen
89142d1e8d removed (not all) warnings 2012-05-16 13:42:32 +02:00
Roland 'Quix0r' Haeder
a093ccf5eb Now used synchronization in all close() methods to make sure all objects 
are 'closed' in an ordered way

Conflicts:
	source/de/anomic/http/server/ChunkedInputStream.java
	source/de/anomic/http/server/ChunkedOutputStream.java
	source/de/anomic/http/server/ContentLengthInputStream.java
	source/net/yacy/cora/protocol/Domains.java
	source/net/yacy/cora/services/federated/solr/SolrShardingConnector.java
	source/net/yacy/cora/services/federated/solr/SolrSingleConnector.java
	source/net/yacy/document/content/dao/PhpBB3Dao.java
	source/net/yacy/document/parser/html/AbstractTransformer.java
	source/net/yacy/kelondro/blob/BEncodedHeap.java
	source/net/yacy/kelondro/blob/HeapReader.java
	source/net/yacy/kelondro/index/RAMIndexCluster.java
	source/net/yacy/kelondro/io/ByteCountInputStream.java
	source/net/yacy/kelondro/logging/ConsoleOutErrHandler.java
	source/net/yacy/kelondro/table/SQLTable.java
 2012-05-14 07:41:55 +02:00
Michael Peter Christen
87f0210480 enriched log output to find NPE in HeapReader 2012-01-15 12:08:46 +01:00
orbiter
5a55397f99 some last-minute performance hacks 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@8101 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2011-11-25 11:23:52 +00:00
orbiter
7598a9e26b fix for thread dump 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7992 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2011-10-07 23:23:49 +00:00
orbiter
8eef8722d1 update to ThreadDump analysis: freerunner and thread state recognition 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7990 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2011-10-07 22:53:14 +00:00
orbiter
57d5529a01 performance hacks 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7977 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2011-09-28 21:16:40 +00:00
orbiter
31283ecd07 - added a search option to filter only specific network protocols. i.e. get only results from ftp servers. Just add '/ftp' to your search. 
for example search for "passwd /ftp". This can also be done with /http /https and /smb
- fixed some search throttling processes that should protect your peer against search DoS or strong search load

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7794 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2011-06-23 11:57:17 +00:00
orbiter
7db208c992 performance hacks: more pre-allocated StringBuilder 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7790 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2011-06-21 23:10:50 +00:00
orbiter
d27a0a67ff fix in log initialization according to hint from Dominic 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7728 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2011-05-17 15:53:59 +00:00
orbiter
2b5f8585bf performance hack for Balancer and ip address parsing 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7608 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2011-03-17 21:09:18 +00:00
orbiter
b1d133b69f another anhancement to the ThreadDump function: better multiple dumps and filtering out of not interesting dump parts 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7606 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2011-03-17 20:48:39 +00:00
orbiter
859c99886c fix for multiple thread dump 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7603 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2011-03-15 23:05:51 +00:00
orbiter
c2a968c23f fix for bug in formatting in ThreadDump 
and added hint for linux/Mac users that they may use the LOCKED feature using the start option -l

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7601 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2011-03-15 08:39:05 +00:00
orbiter
0324de1467 removed debug line 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7597 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2011-03-14 21:34:42 +00:00
orbiter
1aba7869bf patch for Windows: do not use the thread lock feature from previous commit if used on Windows 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7596 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2011-03-14 21:33:36 +00:00
orbiter
0a11727374 added new feature for Thread dump: 
"THREADS WITH STATES: LOCK FOR OTHERS"
will show only such threads that lock other threads. This is the 'opposite part' of the blocked threads.
Because that this uses a thread dump that is produced with a kill -3 on the PID of the process and such thread dumps are written by the Java core outside of System.out and Sytem.err it is necessary to read the dump from a log in the file system. Such a log is only written if YaCy is started with startYACY.sh on a linux system. That means:
this feature is only available on linux and Mac OS X if YaCy is started with ./startYACY.sh -l


git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7595 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2011-03-14 21:32:20 +00:00
orbiter
e6c3507b17 disabled some of the previous changes (did not work in openjdk) 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7591 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2011-03-14 20:48:36 +00:00
orbiter
f9e5c21083 update to thread dump logs 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7590 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2011-03-14 20:46:04 +00:00
orbiter
96bb33ed9b added default size to StringBuffer in logger (and it is not possible to replace the StringBuffer with a StringBuilder...) 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7573 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2011-03-09 09:53:57 +00:00
orbiter
e1b6916423 always try to guess the size of a StringBuilder to prevent too many memory re-allocations 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7572 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2011-03-09 09:29:05 +00:00
low012
bf27a72d53 *) set SVN properties 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7564 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2011-03-07 23:05:23 +00:00