Commit Graph

9 Commits

Author SHA1 Message Date
theli
a2fa75e688 *) Asynchronous queuing of crawl job URLs (stackCrawl)
various checks like the blacklist check or the robots.txt disallow check are now
   done by a separate thread to unburden the indexer thread(s)
   TODO: maybe we have to introduce a threadpool here if it turn out that this single
         thread is a bottleneck because of the time consuming robots.txt downloads

*) improved index transfer
   The index selection and transmission is done in parallel now to improve index 
   transfer performance.
   TODO: maybe we could speed up performance by unsing multiple transmission threads in 
         parallel instead of only a single one.

*) gzip encoded post requests
   it is now configureable if a gzip encoded post request should be send on
   intex transfer/distribution

*) storage Peer (very experimentell and not optimized yet)
   Now it's possible to send the result of the yacy indexer thread to a remote peer 
   istead of storing the indexed words locally. 
   This could be done by setting the property "storagePeerHash" in the yacy config file
   - Please note that if the index transfer fails, the index ist stored locally.
   - TODO: currently this index transfer is done by the indexer thread. 
     To seedup the indexer
     a) this transmission should be done in parallel and
     b) multiple chunks should be bundled and transfered together


*) general performance improvements  
   - better memory cleanup after http request processing has finished
   - replacing some string concatenations with stringBuffers
   - replacing BufferedInputStreams with serverByteBuffer
   - replacing vectors with arraylists wherever possible
   - replacing hashtables with hashmaps wherever possible
   This was done because function calls to verctor or hashtable functions
   take 3 time longer than calls to functions of arraylists or hashmaps.
   TODO: we should take a look on the class serverObject which is inherited from hashmap
         Do we realy need a synchronization for this class?
   TODO: replace arraylists with linkedLists if random access to the list elements is not needed

*) Robots Parser supports if-modified-since downloads now
   If the downloaded robots.txt file is older than 7 days the robots parser tries to
   download the robots.txt with the if-modified-since header to avoid unnecessary downloads
   if the file was not changed. Additionally the ETag header is used to detect changes.

*) Crawler: better handling of unsupported mimeTypes + FileExtension

*) Bugfix: plasmaWordIndexEntity was not closed correctly in 
   - query.java
   - plasmaswitchboard.java

*) function minimizeUrlDB added to yacy.java 
   this function tests the current urlHashDB for unused urls
   ATTENTION: please don't use this function at the moment because
              it causes the wordIndexDB to flush all words into the
              word directory!

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@853 6c8d7289-2bf4-0310-a012-ef5d649a1542
2005-10-05 10:45:33 +00:00
allo
0d912a3de9 removed not needed copyFile Method.
(same exists in Skins_p.java (there its used))


git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@585 6c8d7289-2bf4-0310-a012-ef5d649a1542
2005-08-24 13:05:47 +00:00
allo
a223faace1 not recursive, but it should work on Windows.
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@516 6c8d7289-2bf4-0310-a012-ef5d649a1542
2005-08-11 09:23:46 +00:00
allo
ee0a9a2d9b recursive Translations.
You can now translate the Menu and other things in subfolders, too


git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@508 6c8d7289-2bf4-0310-a012-ef5d649a1542
2005-08-09 14:27:18 +00:00
allo
8290969150 Automatic regeneration of Locales on Restart with a new (SVN) Version
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@353 6c8d7289-2bf4-0310-a012-ef5d649a1542
2005-06-30 20:30:05 +00:00
allo
129929b396 Preparations for automatic Languagefile upgrade on new YaCy Version.
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@352 6c8d7289-2bf4-0310-a012-ef5d649a1542
2005-06-30 19:39:19 +00:00
allo
bef3aaec38 better Translationsystem.
You only need to escape Chars, which have special meanings in regexps.
some you have to escape: ()[]{}


git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@348 6c8d7289-2bf4-0310-a012-ef5d649a1542
2005-06-30 15:28:40 +00:00
orbiter
650ca3955a added flush-thread for index cache and added language-name mapping in Language_p
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@203 6c8d7289-2bf4-0310-a012-ef5d649a1542
2005-05-31 17:39:14 +00:00
allo
a8b2dccc7a New Translation System.
Use Wordlists like these:
de.lng
Network\ Overview=Netzwerk-Übersicht


git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@176 6c8d7289-2bf4-0310-a012-ef5d649a1542
2005-05-26 10:26:32 +00:00