Commit Graph

565 Commits

Author SHA1 Message Date
orbiter
1aba7869bf patch for Windows: do not use the thread lock feature from previous commit if used on Windows
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7596 6c8d7289-2bf4-0310-a012-ef5d649a1542
2011-03-14 21:33:36 +00:00
orbiter
0a11727374 added new feature for Thread dump:
"THREADS WITH STATES: LOCK FOR OTHERS"
will show only such threads that lock other threads. This is the 'opposite part' of the blocked threads.
Because that this uses a thread dump that is produced with a kill -3 on the PID of the process and such thread dumps are written by the Java core outside of System.out and Sytem.err it is necessary to read the dump from a log in the file system. Such a log is only written if YaCy is started with startYACY.sh on a linux system. That means:
this feature is only available on linux and Mac OS X if YaCy is started with ./startYACY.sh -l


git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7595 6c8d7289-2bf4-0310-a012-ef5d649a1542
2011-03-14 21:32:20 +00:00
orbiter
b62b79675b removed type cast warnings
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7594 6c8d7289-2bf4-0310-a012-ef5d649a1542
2011-03-14 21:08:18 +00:00
orbiter
a07a1a8b1e removed type cast warnings
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7593 6c8d7289-2bf4-0310-a012-ef5d649a1542
2011-03-14 21:07:15 +00:00
orbiter
8edaccfedf removed unused variables
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7592 6c8d7289-2bf4-0310-a012-ef5d649a1542
2011-03-14 21:03:37 +00:00
orbiter
e6c3507b17 disabled some of the previous changes (did not work in openjdk)
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7591 6c8d7289-2bf4-0310-a012-ef5d649a1542
2011-03-14 20:48:36 +00:00
orbiter
f9e5c21083 update to thread dump logs
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7590 6c8d7289-2bf4-0310-a012-ef5d649a1542
2011-03-14 20:46:04 +00:00
orbiter
8f11d3a5bb redesigned the ScoreMap classes:
- new concurrent score map using atom operation from java concurrency classes
- redesigned difference beween StaticScore and Dynamic Score into ScoreMap and ReversibleScoreMap allowed that many classes can now use simple ScoreMap Objects which can be used better in concurrent environments using the ConcurrentScoreMap
- switched from DynamicScore to ConcurrentScoreMap usage wherever possible

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7586 6c8d7289-2bf4-0310-a012-ef5d649a1542
2011-03-13 01:41:44 +00:00
orbiter
a564230c48 more enhancements against blocked threads occurred in seed age evaluation (blocks httpd in some cases)
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7585 6c8d7289-2bf4-0310-a012-ef5d649a1542
2011-03-12 22:54:41 +00:00
orbiter
dc0db3550e avoid string conversion
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7584 6c8d7289-2bf4-0310-a012-ef5d649a1542
2011-03-11 00:59:27 +00:00
orbiter
694fa3a2a5 - replaced more direct string-based UTF-8 conversions by predefined UTF-8 conversion
- changed menu structure slightly

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7583 6c8d7289-2bf4-0310-a012-ef5d649a1542
2011-03-10 23:25:07 +00:00
orbiter
30aed9824a moved getBytes() to UTF8.getBytes() to use a default String encoding
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7580 6c8d7289-2bf4-0310-a012-ef5d649a1542
2011-03-10 12:35:32 +00:00
lotus
cb6d307bba adding extension for parser
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7579 6c8d7289-2bf4-0310-a012-ef5d649a1542
2011-03-09 20:36:01 +00:00
orbiter
3820525464 more memory protection: auto-flush of caches in case of memory shortage
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7575 6c8d7289-2bf4-0310-a012-ef5d649a1542
2011-03-09 16:32:34 +00:00
orbiter
7962d35425 - removed file upload function in crawl start and replaced it with an input field for a file path where the crawl start file is loaded. This was necessary to support the API steering for file crawl starts, for two reasons:
1) if the file is changed for a re-crawl this is not reflected in the steering because it would take the previously uploaded crawl start file
2) browsers do not submit the full path of the selected file even if this path is shown in the input field because of security reasons. There is no work-around or hack to make the submission of the full path possible

- fixed deletion of crawl start point urls in crawl stack and balancer double-check
- fixed a problem with steering self-call (no resolving of localhost)
- added more logging for the crawler to supervise why crawl urls are not taken by the loader
- added a javascript onload-function to select domain restriction in all cases where a crawl is started from a file or from a url
- fixed the restrict-to-domain pattern computation, added a 'www.'-prefix and added this functionality also to a crawl start from file 

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7574 6c8d7289-2bf4-0310-a012-ef5d649a1542
2011-03-09 12:50:39 +00:00
orbiter
96bb33ed9b added default size to StringBuffer in logger (and it is not possible to replace the StringBuffer with a StringBuilder...)
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7573 6c8d7289-2bf4-0310-a012-ef5d649a1542
2011-03-09 09:53:57 +00:00
orbiter
e1b6916423 always try to guess the size of a StringBuilder to prevent too many memory re-allocations
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7572 6c8d7289-2bf4-0310-a012-ef5d649a1542
2011-03-09 09:29:05 +00:00
low012
3b40b98256 *) set SVN properties
*) minor changes

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7567 6c8d7289-2bf4-0310-a012-ef5d649a1542
2011-03-08 01:51:51 +00:00
orbiter
619b561a4a enhanced secondary search: index abstracts decompression is now much faster and does not cause strong CPU load after several searches with more than one word
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7565 6c8d7289-2bf4-0310-a012-ef5d649a1542
2011-03-07 23:12:39 +00:00
low012
bf27a72d53 *) set SVN properties
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7564 6c8d7289-2bf4-0310-a012-ef5d649a1542
2011-03-07 23:05:23 +00:00
low012
b649ce2dd7 *) minor changes
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7563 6c8d7289-2bf4-0310-a012-ef5d649a1542
2011-03-07 22:59:19 +00:00
orbiter
70a996a06c reverted SVN 7557 because these classes are called using reflection. The class declaration is in the log configuration. Without these classes you get errors during runtime and a non-formatted log output, i.e.:
STARTUP: Trying to load logging configuration from file /Data/workspace1/yacy/DATA/LOG/yacy.logging
Can't load log handler "net.yacy.kelondro.logging.ConsoleOutErrHandler"
java.lang.ClassNotFoundException: net.yacy.kelondro.logging.ConsoleOutErrHandler
java.lang.ClassNotFoundException: net.yacy.kelondro.logging.ConsoleOutErrHandler
	at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
	at java.security.AccessController.doPrivileged(Native Method)
	at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
	at java.lang.ClassLoader.loadClass(ClassLoader.java:307)
	at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
	at java.lang.ClassLoader.loadClass(ClassLoader.java:248)
	at java.util.logging.LogManager$3.run(LogManager.java:359)
	at java.security.AccessController.doPrivileged(Native Method)
	at java.util.logging.LogManager.loadLoggerHandlers(LogManager.java:346)
	at java.util.logging.LogManager.initializeGlobalHandlers(LogManager.java:898)
	at java.util.logging.LogManager.access$900(LogManager.java:130)
	at java.util.logging.LogManager$RootLogger.getHandlers(LogManager.java:979)
	at java.util.logging.Logger.log(Logger.java:454)
	at java.util.logging.Logger.doLog(Logger.java:480)
	at java.util.logging.Logger.log(Logger.java:503)
	at net.yacy.kelondro.logging.Log$logRunner.run(Log.java:332)
Can't load log handler "net.yacy.kelondro.logging.LogalizerHandler"
java.lang.ClassNotFoundException: net.yacy.kelondro.logging.LogalizerHandler
java.lang.ClassNotFoundException: net.yacy.kelondro.logging.LogalizerHandler
	at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
	at java.security.AccessController.doPrivileged(Native Method)
	at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
	at java.lang.ClassLoader.loadClass(ClassLoader.java:307)
	at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
	at java.lang.ClassLoader.loadClass(ClassLoader.java:248)
	at java.util.logging.LogManager$3.run(LogManager.java:359)
	at java.security.AccessController.doPrivileged(Native Method)
	at java.util.logging.LogManager.loadLoggerHandlers(LogManager.java:346)
	at java.util.logging.LogManager.initializeGlobalHandlers(LogManager.java:898)
	at java.util.logging.LogManager.access$900(LogManager.java:130)
	at java.util.logging.LogManager$RootLogger.getHandlers(LogManager.java:979)
	at java.util.logging.Logger.log(Logger.java:454)
	at java.util.logging.Logger.doLog(Logger.java:480)
	at java.util.logging.Logger.log(Logger.java:503)
	at net.yacy.kelondro.logging.Log$logRunner.run(Log.java:332)

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7559 6c8d7289-2bf4-0310-a012-ef5d649a1542
2011-03-07 20:42:19 +00:00
orbiter
cb1f49d0f2 replaced all 'new String' with default encoding (missing) or UTF-8 encoding with a String generation method that uses a pre-defined Charset constant for UTF-8. This avoids a cache-lookup for the Charset object using String hashing of the String 'UTF-8'.
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7558 6c8d7289-2bf4-0310-a012-ef5d649a1542
2011-03-07 20:36:40 +00:00
low012
9d366ee9d7 *) removed unused code (I assume that most of the code was really dead, but if you need any of the classes, tell me and I will put it back in.)
*) minor code cleanup in ViewLog

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7557 6c8d7289-2bf4-0310-a012-ef5d649a1542
2011-03-07 18:55:11 +00:00
orbiter
7138f4036b less synchronization, better thread dump tool
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7556 6c8d7289-2bf4-0310-a012-ef5d649a1542
2011-03-07 15:29:45 +00:00
orbiter
8d14916c74 more patches for a better out-of-memory management
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7555 6c8d7289-2bf4-0310-a012-ef5d649a1542
2011-03-07 01:45:11 +00:00
orbiter
c2c5b12882 - even less memory for circle tool
- background thread for bookmark initialization: this uses a DNS lookup which may cause long waiting times during startup

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7554 6c8d7289-2bf4-0310-a012-ef5d649a1542
2011-03-06 12:30:22 +00:00
orbiter
6badc5e558 reduce size of static memory usage: use short instead of int in circle coordinates cache
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7553 6c8d7289-2bf4-0310-a012-ef5d649a1542
2011-03-06 11:23:04 +00:00
orbiter
ce0c8247fc removed (most probably!?!) superfluos System.err output
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7552 6c8d7289-2bf4-0310-a012-ef5d649a1542
2011-03-06 01:01:29 +00:00
orbiter
799c534935 one more patch again OOM during secondary remote search
see also: http://forum.yacy-websuche.de/viewtopic.php?f=6&t=3202

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7551 6c8d7289-2bf4-0310-a012-ef5d649a1542
2011-03-05 19:52:34 +00:00
orbiter
f8d0454c53 small bug fixes and experiments with search speed enhancement
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7549 6c8d7289-2bf4-0310-a012-ef5d649a1542
2011-03-04 14:29:22 +00:00
orbiter
993b9bc1a8 memory/performance hacks, less synchronization, better concurrency
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7544 6c8d7289-2bf4-0310-a012-ef5d649a1542
2011-03-03 11:30:04 +00:00
sixcooler
65bcc60808 stupid me: revert placement of closing connection which caused unclosed connections
+ reuse sockets

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7543 6c8d7289-2bf4-0310-a012-ef5d649a1542
2011-03-03 04:16:33 +00:00
sixcooler
e3d75d6cd5 Not storing external header in an Header-Array and reduce a loop for its conversion.
Ensure connection close if a OOM is thrown.
Ensure setting resolved host is set at the request.

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7542 6c8d7289-2bf4-0310-a012-ef5d649a1542
2011-03-01 13:50:11 +00:00
orbiter
42d90664f3 - fixed a memory leak in the httpc.post method (no finish)
- patched some more memory-saving relevant code
- some more minor bug fixes

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7541 6c8d7289-2bf4-0310-a012-ef5d649a1542
2011-03-01 09:03:33 +00:00
orbiter
38dce547c0 better concurrency (less locking on date formatting) more logging and minor bug fixes
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7540 6c8d7289-2bf4-0310-a012-ef5d649a1542
2011-02-28 06:28:29 +00:00
orbiter
89d337841c more logging for OOMs
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7534 6c8d7289-2bf4-0310-a012-ef5d649a1542
2011-02-26 09:27:15 +00:00
orbiter
b1781d7aae some more performance hacks
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7533 6c8d7289-2bf4-0310-a012-ef5d649a1542
2011-02-26 01:24:49 +00:00
orbiter
b2f147d28e performance hack: excluded map encoding in many cases from synchronization block, especially when doing an iteration
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7532 6c8d7289-2bf4-0310-a012-ef5d649a1542
2011-02-25 21:29:55 +00:00
orbiter
5e186e0122 continuing the fight against deadlocks during time formatting: better caching.
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7531 6c8d7289-2bf4-0310-a012-ef5d649a1542
2011-02-25 21:11:53 +00:00
orbiter
dec24244cf added convenience class to generate UTF StringBody objects with a default UTF8 charset.
Reason: if this is not used in StringBody-Class initialization, a default charset name is parsed.
This is a synchronized process and all classes using default charsets synchronize at that point
Synchronization is omitted if this class is used
 

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7530 6c8d7289-2bf4-0310-a012-ef5d649a1542
2011-02-25 13:26:09 +00:00
orbiter
1110d16af9 performance hack: replaced generic row.getColBytes() call with row.getPrimaryKeyBytes() where the column is 0
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7529 6c8d7289-2bf4-0310-a012-ef5d649a1542
2011-02-25 12:41:27 +00:00
orbiter
19b2a50578 - enhanced date formatter cache
- added more instances of formatter objects to different classes to make them independent in case of lockings that may applay during synchronization of the date formatter object (date formatting is not thread-safe and must be synchronized therefore)

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7528 6c8d7289-2bf4-0310-a012-ef5d649a1542
2011-02-25 12:23:00 +00:00
orbiter
48a61c39a3 speed hacks in BLOB ArrayStack:
- more concurrency if possible
- less threads if no concurrency necessary

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7527 6c8d7289-2bf4-0310-a012-ef5d649a1542
2011-02-25 11:58:01 +00:00
orbiter
a92d80a545 performance enhancements using an alternative to a insensitive collator (a complex string compare):
- less synchronizations
- better speed
..at most important and commonly used classes: http headers, url parsing and html parsing

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7526 6c8d7289-2bf4-0310-a012-ef5d649a1542
2011-02-25 11:23:57 +00:00
sixcooler
bcea497644 next try to fix for http://forum.yacy-websuche.de/viewtopic.php?f=6&t=3193&start=0&sid=b98aa9a7466397602b436eb45f4a9d39
tested proxy, crawl, updatedownload - please do further testing!

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7524 6c8d7289-2bf4-0310-a012-ef5d649a1542
2011-02-25 01:16:15 +00:00
orbiter
ad7fcb9d61 Enhanced Base64Order transformation: less overhead (transformation between StringBuilder and byte[])
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7523 6c8d7289-2bf4-0310-a012-ef5d649a1542
2011-02-25 00:56:31 +00:00
orbiter
f95e50ec3d more explanation
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7522 6c8d7289-2bf4-0310-a012-ef5d649a1542
2011-02-25 00:06:31 +00:00
orbiter
bb36bf841a emergency commit (sorry sixcooler for not waiting) because without that automatic updating peers would not be able to do the next update.
Please see http://forum.yacy-websuche.de/viewtopic.php?p=22059#p22059

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7521 6c8d7289-2bf4-0310-a012-ef5d649a1542
2011-02-25 00:02:04 +00:00
sixcooler
8ad4e10491 fix for http://forum.yacy-websuche.de/viewtopic.php?f=6&t=3193&start=0&sid=b98aa9a7466397602b436eb45f4a9d39
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7520 6c8d7289-2bf4-0310-a012-ef5d649a1542
2011-02-24 23:15:27 +00:00