Commit Graph

7503 Commits

Author SHA1 Message Date
suessthomas
9956dc9dce Update jcifs-library to Version 1.3.15. Small Changes, read: http://jcifs.samba.org/ - "Minor adjustments have been applied to DcerpcHandle locking routines in the SID class to fix sporadic occurances of "All pipe instances are busy" errors under high load."
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7587 6c8d7289-2bf4-0310-a012-ef5d649a1542
2011-03-13 20:09:52 +00:00
orbiter
8f11d3a5bb redesigned the ScoreMap classes:
- new concurrent score map using atom operation from java concurrency classes
- redesigned difference beween StaticScore and Dynamic Score into ScoreMap and ReversibleScoreMap allowed that many classes can now use simple ScoreMap Objects which can be used better in concurrent environments using the ConcurrentScoreMap
- switched from DynamicScore to ConcurrentScoreMap usage wherever possible

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7586 6c8d7289-2bf4-0310-a012-ef5d649a1542
2011-03-13 01:41:44 +00:00
orbiter
a564230c48 more enhancements against blocked threads occurred in seed age evaluation (blocks httpd in some cases)
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7585 6c8d7289-2bf4-0310-a012-ef5d649a1542
2011-03-12 22:54:41 +00:00
orbiter
dc0db3550e avoid string conversion
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7584 6c8d7289-2bf4-0310-a012-ef5d649a1542
2011-03-11 00:59:27 +00:00
orbiter
694fa3a2a5 - replaced more direct string-based UTF-8 conversions by predefined UTF-8 conversion
- changed menu structure slightly

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7583 6c8d7289-2bf4-0310-a012-ef5d649a1542
2011-03-10 23:25:07 +00:00
lotus
bbb7aea8f3 fix basic config change in portal mode
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7582 6c8d7289-2bf4-0310-a012-ef5d649a1542
2011-03-10 20:04:15 +00:00
pca
df68bf6001 Windows Installer:
- check Windows-Version on startup, support only Windows 2000 and newer (necessary for Sun-JRE and as preparation for firewall section)
- little changes in JRE section handling

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7581 6c8d7289-2bf4-0310-a012-ef5d649a1542
2011-03-10 19:30:05 +00:00
orbiter
30aed9824a moved getBytes() to UTF8.getBytes() to use a default String encoding
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7580 6c8d7289-2bf4-0310-a012-ef5d649a1542
2011-03-10 12:35:32 +00:00
lotus
cb6d307bba adding extension for parser
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7579 6c8d7289-2bf4-0310-a012-ef5d649a1542
2011-03-09 20:36:01 +00:00
pca
61a64bdbef Windows Installer:
- detect JRE at startup, showing install-option depends on result
- hide window for external call "attrib"
- some cleanup and restructure for readability

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7578 6c8d7289-2bf4-0310-a012-ef5d649a1542
2011-03-09 20:19:34 +00:00
orbiter
4d733608fb fix for broken JSON, see: http://forum.yacy-websuche.de/viewtopic.php?p=22162#p22162
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7577 6c8d7289-2bf4-0310-a012-ef5d649a1542
2011-03-09 20:08:20 +00:00
orbiter
1214615185 fix for 'invisible entry', see http://forum.yacy-websuche.de/viewtopic.php?p=22133#p22133
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7576 6c8d7289-2bf4-0310-a012-ef5d649a1542
2011-03-09 17:04:34 +00:00
orbiter
3820525464 more memory protection: auto-flush of caches in case of memory shortage
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7575 6c8d7289-2bf4-0310-a012-ef5d649a1542
2011-03-09 16:32:34 +00:00
orbiter
7962d35425 - removed file upload function in crawl start and replaced it with an input field for a file path where the crawl start file is loaded. This was necessary to support the API steering for file crawl starts, for two reasons:
1) if the file is changed for a re-crawl this is not reflected in the steering because it would take the previously uploaded crawl start file
2) browsers do not submit the full path of the selected file even if this path is shown in the input field because of security reasons. There is no work-around or hack to make the submission of the full path possible

- fixed deletion of crawl start point urls in crawl stack and balancer double-check
- fixed a problem with steering self-call (no resolving of localhost)
- added more logging for the crawler to supervise why crawl urls are not taken by the loader
- added a javascript onload-function to select domain restriction in all cases where a crawl is started from a file or from a url
- fixed the restrict-to-domain pattern computation, added a 'www.'-prefix and added this functionality also to a crawl start from file 

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7574 6c8d7289-2bf4-0310-a012-ef5d649a1542
2011-03-09 12:50:39 +00:00
orbiter
96bb33ed9b added default size to StringBuffer in logger (and it is not possible to replace the StringBuffer with a StringBuilder...)
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7573 6c8d7289-2bf4-0310-a012-ef5d649a1542
2011-03-09 09:53:57 +00:00
orbiter
e1b6916423 always try to guess the size of a StringBuilder to prevent too many memory re-allocations
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7572 6c8d7289-2bf4-0310-a012-ef5d649a1542
2011-03-09 09:29:05 +00:00
low012
bea8137997 *) minor changes
*) fixed potential NPE in suggest.java

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7571 6c8d7289-2bf4-0310-a012-ef5d649a1542
2011-03-08 23:27:41 +00:00
low012
3e03963b1c *) minor changes
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7570 6c8d7289-2bf4-0310-a012-ef5d649a1542
2011-03-08 22:37:17 +00:00
pca
8b94e33a33 Cleanup for last commit (forget some unnecessary parts)
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7569 6c8d7289-2bf4-0310-a012-ef5d649a1542
2011-03-08 19:11:49 +00:00
pca
b49f8894e1 Add check for hard drive and free space to Windows-Installer
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7568 6c8d7289-2bf4-0310-a012-ef5d649a1542
2011-03-08 19:07:22 +00:00
low012
3b40b98256 *) set SVN properties
*) minor changes

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7567 6c8d7289-2bf4-0310-a012-ef5d649a1542
2011-03-08 01:51:51 +00:00
orbiter
2af8e33773 better performance computing search targets with index abstracts
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7566 6c8d7289-2bf4-0310-a012-ef5d649a1542
2011-03-07 23:32:01 +00:00
orbiter
619b561a4a enhanced secondary search: index abstracts decompression is now much faster and does not cause strong CPU load after several searches with more than one word
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7565 6c8d7289-2bf4-0310-a012-ef5d649a1542
2011-03-07 23:12:39 +00:00
low012
bf27a72d53 *) set SVN properties
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7564 6c8d7289-2bf4-0310-a012-ef5d649a1542
2011-03-07 23:05:23 +00:00
low012
b649ce2dd7 *) minor changes
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7563 6c8d7289-2bf4-0310-a012-ef5d649a1542
2011-03-07 22:59:19 +00:00
low012
0da3b6489e *) added the only changes from r7557 which actualy made sense
*) caught potential exception (occured when user entered a string which did not contain digits only for the maximum number of lines)
*) use prop.putHTML to avoid potential XSS attack in case an attacker manages to cause something to end up in the logs which contains a string which was defined by the attacker

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7562 6c8d7289-2bf4-0310-a012-ef5d649a1542
2011-03-07 21:44:58 +00:00
orbiter
27ecdb5444 use less peers for remote search
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7561 6c8d7289-2bf4-0310-a012-ef5d649a1542
2011-03-07 21:24:46 +00:00
orbiter
29acd2f108 reverted also changes in ViewLog from SVN 7557 because the ThreadDump submenu was not visible any more.
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7560 6c8d7289-2bf4-0310-a012-ef5d649a1542
2011-03-07 21:19:47 +00:00
orbiter
70a996a06c reverted SVN 7557 because these classes are called using reflection. The class declaration is in the log configuration. Without these classes you get errors during runtime and a non-formatted log output, i.e.:
STARTUP: Trying to load logging configuration from file /Data/workspace1/yacy/DATA/LOG/yacy.logging
Can't load log handler "net.yacy.kelondro.logging.ConsoleOutErrHandler"
java.lang.ClassNotFoundException: net.yacy.kelondro.logging.ConsoleOutErrHandler
java.lang.ClassNotFoundException: net.yacy.kelondro.logging.ConsoleOutErrHandler
	at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
	at java.security.AccessController.doPrivileged(Native Method)
	at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
	at java.lang.ClassLoader.loadClass(ClassLoader.java:307)
	at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
	at java.lang.ClassLoader.loadClass(ClassLoader.java:248)
	at java.util.logging.LogManager$3.run(LogManager.java:359)
	at java.security.AccessController.doPrivileged(Native Method)
	at java.util.logging.LogManager.loadLoggerHandlers(LogManager.java:346)
	at java.util.logging.LogManager.initializeGlobalHandlers(LogManager.java:898)
	at java.util.logging.LogManager.access$900(LogManager.java:130)
	at java.util.logging.LogManager$RootLogger.getHandlers(LogManager.java:979)
	at java.util.logging.Logger.log(Logger.java:454)
	at java.util.logging.Logger.doLog(Logger.java:480)
	at java.util.logging.Logger.log(Logger.java:503)
	at net.yacy.kelondro.logging.Log$logRunner.run(Log.java:332)
Can't load log handler "net.yacy.kelondro.logging.LogalizerHandler"
java.lang.ClassNotFoundException: net.yacy.kelondro.logging.LogalizerHandler
java.lang.ClassNotFoundException: net.yacy.kelondro.logging.LogalizerHandler
	at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
	at java.security.AccessController.doPrivileged(Native Method)
	at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
	at java.lang.ClassLoader.loadClass(ClassLoader.java:307)
	at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
	at java.lang.ClassLoader.loadClass(ClassLoader.java:248)
	at java.util.logging.LogManager$3.run(LogManager.java:359)
	at java.security.AccessController.doPrivileged(Native Method)
	at java.util.logging.LogManager.loadLoggerHandlers(LogManager.java:346)
	at java.util.logging.LogManager.initializeGlobalHandlers(LogManager.java:898)
	at java.util.logging.LogManager.access$900(LogManager.java:130)
	at java.util.logging.LogManager$RootLogger.getHandlers(LogManager.java:979)
	at java.util.logging.Logger.log(Logger.java:454)
	at java.util.logging.Logger.doLog(Logger.java:480)
	at java.util.logging.Logger.log(Logger.java:503)
	at net.yacy.kelondro.logging.Log$logRunner.run(Log.java:332)

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7559 6c8d7289-2bf4-0310-a012-ef5d649a1542
2011-03-07 20:42:19 +00:00
orbiter
cb1f49d0f2 replaced all 'new String' with default encoding (missing) or UTF-8 encoding with a String generation method that uses a pre-defined Charset constant for UTF-8. This avoids a cache-lookup for the Charset object using String hashing of the String 'UTF-8'.
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7558 6c8d7289-2bf4-0310-a012-ef5d649a1542
2011-03-07 20:36:40 +00:00
low012
9d366ee9d7 *) removed unused code (I assume that most of the code was really dead, but if you need any of the classes, tell me and I will put it back in.)
*) minor code cleanup in ViewLog

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7557 6c8d7289-2bf4-0310-a012-ef5d649a1542
2011-03-07 18:55:11 +00:00
orbiter
7138f4036b less synchronization, better thread dump tool
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7556 6c8d7289-2bf4-0310-a012-ef5d649a1542
2011-03-07 15:29:45 +00:00
orbiter
8d14916c74 more patches for a better out-of-memory management
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7555 6c8d7289-2bf4-0310-a012-ef5d649a1542
2011-03-07 01:45:11 +00:00
orbiter
c2c5b12882 - even less memory for circle tool
- background thread for bookmark initialization: this uses a DNS lookup which may cause long waiting times during startup

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7554 6c8d7289-2bf4-0310-a012-ef5d649a1542
2011-03-06 12:30:22 +00:00
orbiter
6badc5e558 reduce size of static memory usage: use short instead of int in circle coordinates cache
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7553 6c8d7289-2bf4-0310-a012-ef5d649a1542
2011-03-06 11:23:04 +00:00
orbiter
ce0c8247fc removed (most probably!?!) superfluos System.err output
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7552 6c8d7289-2bf4-0310-a012-ef5d649a1542
2011-03-06 01:01:29 +00:00
orbiter
799c534935 one more patch again OOM during secondary remote search
see also: http://forum.yacy-websuche.de/viewtopic.php?f=6&t=3202

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7551 6c8d7289-2bf4-0310-a012-ef5d649a1542
2011-03-05 19:52:34 +00:00
orbiter
77b1e921a9 this asserts prevents a network operation in case of sabotage and must be removed therefore
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7550 6c8d7289-2bf4-0310-a012-ef5d649a1542
2011-03-04 14:34:37 +00:00
orbiter
f8d0454c53 small bug fixes and experiments with search speed enhancement
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7549 6c8d7289-2bf4-0310-a012-ef5d649a1542
2011-03-04 14:29:22 +00:00
orbiter
bed79402be introduction of a new remote search load control: the remote search has taken 10 results per peer with a time-out of 3 seconds so far. The attributes of number of results per peer and time-out time can now be configured.
This has two aspects: the user who searches may want to increase these values to get more results and more load on the remote side and the user of the server which is accessed for this search may want to restrict the load. Both sides can now be configured. The server-site maximum load parameters are defined by a network definition and the client-side search request load can be defined by each user individually but when the remote search is done the requested service is limited to the network definition.

You can find now in the network definition file:
network.unit.remotesearch.maxcount and network.unit.remotesearch.maxtime
and in the yacy.conf file:
remotesearch.maxcount and remotesearch.maxtime

There is currently no web interface to define the client-side remote search attributes, please set them manually
    

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7548 6c8d7289-2bf4-0310-a012-ef5d649a1542
2011-03-04 13:44:00 +00:00
orbiter
6dfaf6fef7 fix for bug in deletion of old seeds
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7547 6c8d7289-2bf4-0310-a012-ef5d649a1542
2011-03-04 10:00:37 +00:00
lotus
ac13343f12 jre update
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7546 6c8d7289-2bf4-0310-a012-ef5d649a1542
2011-03-03 20:44:53 +00:00
lotus
fa5b911082 remove old libraries in case of update
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7545 6c8d7289-2bf4-0310-a012-ef5d649a1542
2011-03-03 18:34:32 +00:00
orbiter
993b9bc1a8 memory/performance hacks, less synchronization, better concurrency
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7544 6c8d7289-2bf4-0310-a012-ef5d649a1542
2011-03-03 11:30:04 +00:00
sixcooler
65bcc60808 stupid me: revert placement of closing connection which caused unclosed connections
+ reuse sockets

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7543 6c8d7289-2bf4-0310-a012-ef5d649a1542
2011-03-03 04:16:33 +00:00
sixcooler
e3d75d6cd5 Not storing external header in an Header-Array and reduce a loop for its conversion.
Ensure connection close if a OOM is thrown.
Ensure setting resolved host is set at the request.

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7542 6c8d7289-2bf4-0310-a012-ef5d649a1542
2011-03-01 13:50:11 +00:00
orbiter
42d90664f3 - fixed a memory leak in the httpc.post method (no finish)
- patched some more memory-saving relevant code
- some more minor bug fixes

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7541 6c8d7289-2bf4-0310-a012-ef5d649a1542
2011-03-01 09:03:33 +00:00
orbiter
38dce547c0 better concurrency (less locking on date formatting) more logging and minor bug fixes
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7540 6c8d7289-2bf4-0310-a012-ef5d649a1542
2011-02-28 06:28:29 +00:00
mikeworks
85f5c02deb de.lng: Updated German translation and removed old unused strings, e.g. 8080 -> 8090 and Search Portal translations
Bookmarks.html, Ranking_p.html, base.css: Fixed XHTML errors to make pages compatible again - switched div -> span inside dt and replaced css definition of id (unique) with class (class of elements)
header.template: Fixed link to berliOS changelog by replacing & -> & and adding translation for German page by refering to German berliOS UI ;-]

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7539 6c8d7289-2bf4-0310-a012-ef5d649a1542
2011-02-28 00:22:05 +00:00
f1ori
59dea3a284 * implement url proxy, a proxy via the url http://peer:port/proxy.html?url=http://domain.tld/path
* enable with proxyURL = true
* could be useful to browse specific pages with proxy or use own improvements in proxy

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7538 6c8d7289-2bf4-0310-a012-ef5d649a1542
2011-02-27 21:39:38 +00:00