Marown/yacy_search_server
1
0
Fork 0
mirror of https://github.com/yacy/yacy_search_server.git synced 2024-09-21 00:00:13 +02:00
Code Issues Actions Packages Projects Releases Wiki Activity
897 Commits 1 Branch 20 Tags 836 MiB
b177a80bb7
Commit Graph

76 Commits

Author SHA1 Message Date
orbiter
371fd67ecf headless awt mode 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@922 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2005-10-12 12:34:08 +00:00
borg-0300
6d93ecf947 Thread.getAllStackTraces() removed, needs java 1.5 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@915 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2005-10-11 22:00:40 +00:00
borg-0300
52036caeac changed restart message 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@913 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2005-10-11 16:31:40 +00:00
borg-0300
68aa215479 cleaned 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@866 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2005-10-05 16:49:08 +00:00
borg-0300
fb27428674 added restart to Status.html 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@863 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2005-10-05 16:35:05 +00:00
theli
a2fa75e688 *) Asynchronous queuing of crawl job URLs (stackCrawl) 
various checks like the blacklist check or the robots.txt disallow check are now
   done by a separate thread to unburden the indexer thread(s)
   TODO: maybe we have to introduce a threadpool here if it turn out that this single
         thread is a bottleneck because of the time consuming robots.txt downloads

*) improved index transfer
   The index selection and transmission is done in parallel now to improve index 
   transfer performance.
   TODO: maybe we could speed up performance by unsing multiple transmission threads in 
         parallel instead of only a single one.

*) gzip encoded post requests
   it is now configureable if a gzip encoded post request should be send on
   intex transfer/distribution

*) storage Peer (very experimentell and not optimized yet)
   Now it's possible to send the result of the yacy indexer thread to a remote peer 
   istead of storing the indexed words locally. 
   This could be done by setting the property "storagePeerHash" in the yacy config file
   - Please note that if the index transfer fails, the index ist stored locally.
   - TODO: currently this index transfer is done by the indexer thread. 
     To seedup the indexer
     a) this transmission should be done in parallel and
     b) multiple chunks should be bundled and transfered together


*) general performance improvements  
   - better memory cleanup after http request processing has finished
   - replacing some string concatenations with stringBuffers
   - replacing BufferedInputStreams with serverByteBuffer
   - replacing vectors with arraylists wherever possible
   - replacing hashtables with hashmaps wherever possible
   This was done because function calls to verctor or hashtable functions
   take 3 time longer than calls to functions of arraylists or hashmaps.
   TODO: we should take a look on the class serverObject which is inherited from hashmap
         Do we realy need a synchronization for this class?
   TODO: replace arraylists with linkedLists if random access to the list elements is not needed

*) Robots Parser supports if-modified-since downloads now
   If the downloaded robots.txt file is older than 7 days the robots parser tries to
   download the robots.txt with the if-modified-since header to avoid unnecessary downloads
   if the file was not changed. Additionally the ETag header is used to detect changes.

*) Crawler: better handling of unsupported mimeTypes + FileExtension

*) Bugfix: plasmaWordIndexEntity was not closed correctly in 
   - query.java
   - plasmaswitchboard.java

*) function minimizeUrlDB added to yacy.java 
   this function tests the current urlHashDB for unused urls
   ATTENTION: please don't use this function at the moment because
              it causes the wordIndexDB to flush all words into the
              word directory!

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@853 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2005-10-05 10:45:33 +00:00
orbiter
7fc822a59b changed handling of time-zones 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@801 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2005-09-27 16:28:55 +00:00
allo
4aa04972ac bugfix 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@777 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2005-09-22 21:02:08 +00:00
allo
7991c05b49 homePath instead if RootPath 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@775 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2005-09-22 20:25:56 +00:00
allo
5bf7d74114 permanent yacy.logging 
see http://www.yacy-forum.de/viewtopic.php?p=10020


git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@773 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2005-09-22 18:54:36 +00:00
orbiter
2f732e32a2 enhancements to memory menue 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@762 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2005-09-21 12:21:01 +00:00
orbiter
b5337a122c some more information about available memory in PerformaceMemory menu 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@759 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2005-09-21 00:12:37 +00:00
borg-0300
e748ba3f6e super(), finals; 
other; cleaned;
Properties;

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@755 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2005-09-20 15:36:22 +00:00
orbiter
fb52a82008 added new performance page for memory settings 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@751 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2005-09-20 10:10:34 +00:00
borg-0300
6d43a4970c small changes 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@631 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2005-09-01 15:27:41 +00:00
theli
4fd5b95b1f *) Renaming Logger function names to reflect the proper Java Logging API Loglevels 
- please use logFine instead of logDebug
   - please use logSevere instead of logFailure and logError
   See: http://www.yacy-forum.de/viewtopic.php?p=8726#8726

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@615 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2005-08-30 21:32:59 +00:00
theli
6adf8a4bde *) Renaming Logger function names to reflect the proper Java Logging API Loglevels 
- please use logFine instead of logDebug
   - please use logFailure instead of logError
   See: http://www.yacy-forum.de/viewtopic.php?p=8726#8726

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@614 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2005-08-30 21:10:39 +00:00
theli
0dfa8b62e2 *) Changing Proxy-Useragent string according to thread http://www.yacy-forum.de/viewtopic.php?p=8183#8183 
A typical useragent string now e.g. looks like: 
   Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.7.10; YaCy 0.401/00602; yacy.net) Gecko/20050716 Firefox/1.0.6

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@607 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2005-08-30 13:41:47 +00:00
orbiter
ba0a486328 moved printStackTrace() to logging 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@539 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2005-08-14 23:35:18 +00:00
allo
a223faace1 not recursive, but it should work on Windows. 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@516 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2005-08-11 09:23:46 +00:00
allo
ee0a9a2d9b recursive Translations. 
You can now translate the Menu and other things in subfolders, too


git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@508 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2005-08-09 14:27:18 +00:00
orbiter
60eaf3dcde fix for notifer.gif appearance 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@506 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2005-08-08 22:35:35 +00:00
theli
e6aced0162 *) Setting higher priority for session threads 
See: http://www.yacy-forum.de/viewtopic.php?p=6120#6120

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@490 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2005-08-03 17:48:11 +00:00
orbiter
2d8557cb10 minor changes 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@487 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2005-08-03 02:02:39 +00:00
jerri
3334546340 Started the quest for in-source documentation with javadoc. 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@483 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2005-08-02 19:40:29 +00:00
orbiter
083c8ddc69 new alert symbols 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@473 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2005-08-02 00:16:19 +00:00
orbiter
b9d18d40cb configuration of proxy idle time in performance menue 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@436 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2005-07-26 15:17:29 +00:00
orbiter
3470a72d48 fixed div by zero, set default delays, fixed release number format and display 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@435 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2005-07-26 11:47:50 +00:00
orbiter
b73557ed2d better assortment monitoring and enhanced profile menue 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@416 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2005-07-20 13:03:41 +00:00
orbiter
1f36bf4dae enhanced assortment capacity; added extended WORDS migration 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@412 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2005-07-20 00:39:06 +00:00
orbiter
9c72b4cdec replaced index dump stack by an dump array and limited url number in assortment ram (prevents too much RAM occupation) 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@406 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2005-07-18 13:32:44 +00:00
orbiter
858cd94299 replaced indexing ram-queue by file-based stack-queue 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@381 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2005-07-06 14:48:41 +00:00
orbiter
fbef7fed81 adopted latestVersion to float handling 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@363 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2005-07-03 12:40:36 +00:00
orbiter
86f2aa8478 fixed seed-load date bug (evaluating server date for age computation) 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@354 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2005-06-30 23:19:08 +00:00
allo
8290969150 Automatic regeneration of Locales on Restart with a new (SVN) Version 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@353 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2005-06-30 20:30:05 +00:00
allo
129929b396 Preparations for automatic Languagefile upgrade on new YaCy Version. 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@352 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2005-06-30 19:39:19 +00:00
orbiter
712fe9ef18 bugfixed utf-8 decoding and parser 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@346 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2005-06-29 22:55:37 +00:00
orbiter
68dc2b0c6b added kelondroArray, the basis for upcoming kelondroHash and some bug fixes 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@311 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2005-06-21 01:17:25 +00:00
orbiter
85075269a6 extended fail-safe memory-managament. prevents too much allocation, too often GC and should help for the 100%CPU-bug 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@303 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2005-06-20 00:46:23 +00:00
orbiter
a25b5b4986 fixed possible memory leak in htmlScraper: be aware that now links can get lost; further work necessary 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@288 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2005-06-16 18:31:28 +00:00
orbiter
f45dc29f35 maintenance 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@279 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2005-06-15 14:59:53 +00:00
orbiter
a5b40923b6 added word migration to assortments (start with 'java -classpath classes yacy -migratewords') 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@278 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2005-06-15 01:22:07 +00:00
orbiter
beb652847e added run-target for ant (needed for NetBeans4) 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@274 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2005-06-13 12:01:58 +00:00
theli
ee9e110366 *) removing old logging configuration properties from yacy.init 
*) serverLog.java logging functions now also accept exceptions als
   additional parameters.
   The Stacktrace of this ecceptions will then be appended to the 
   logging message and can e.g. be viewed on the gui logging page

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@265 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2005-06-10 09:19:24 +00:00
theli
318dfc4636 *) activating new logging framework 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@240 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2005-06-09 09:46:43 +00:00
low012
8c2789b22a to catch is an irregular verb 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@225 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2005-06-09 00:07:25 +00:00
orbiter
4574fa4ce7 bugfixes 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@224 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2005-06-08 15:28:29 +00:00
theli
167ddd5612 *) Displaying "SVN xxx" instead of "@REPL_VERSION@" if yacy is started directly from SVN 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@213 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2005-06-07 08:31:49 +00:00
orbiter
ee0758fe4d bugfixes/empty-dir-deletion/snippet-test-activation 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@212 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2005-06-07 01:05:55 +00:00
allo
3b03d99797 Bugfix: Wrong Path of DATA/LOCALE Dir 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@182 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2005-05-27 14:42:54 +00:00