Marown/yacy_search_server
1
0
Fork 0
mirror of https://github.com/yacy/yacy_search_server.git synced 2024-09-21 00:00:13 +02:00
Code Issues Actions Packages Projects Releases Wiki Activity
834 Commits 1 Branch 20 Tags 836 MiB
0dcfc25d98
Commit Graph

560 Commits

Author SHA1 Message Date
borg-0300
0dcfc25d98 cleaned, finals, Properties 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@862 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2005-10-05 16:25:43 +00:00
borg-0300
4191b21e73 cleaned, finals, Properties 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@858 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2005-10-05 13:30:12 +00:00
orbiter
f6cf3967de fix for compile-bug in svn 583 (Martin guck mal ob das richtig ist: fifo oder filo-stack?) 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@854 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2005-10-05 12:21:30 +00:00
theli
a2fa75e688 *) Asynchronous queuing of crawl job URLs (stackCrawl) 
various checks like the blacklist check or the robots.txt disallow check are now
   done by a separate thread to unburden the indexer thread(s)
   TODO: maybe we have to introduce a threadpool here if it turn out that this single
         thread is a bottleneck because of the time consuming robots.txt downloads

*) improved index transfer
   The index selection and transmission is done in parallel now to improve index 
   transfer performance.
   TODO: maybe we could speed up performance by unsing multiple transmission threads in 
         parallel instead of only a single one.

*) gzip encoded post requests
   it is now configureable if a gzip encoded post request should be send on
   intex transfer/distribution

*) storage Peer (very experimentell and not optimized yet)
   Now it's possible to send the result of the yacy indexer thread to a remote peer 
   istead of storing the indexed words locally. 
   This could be done by setting the property "storagePeerHash" in the yacy config file
   - Please note that if the index transfer fails, the index ist stored locally.
   - TODO: currently this index transfer is done by the indexer thread. 
     To seedup the indexer
     a) this transmission should be done in parallel and
     b) multiple chunks should be bundled and transfered together


*) general performance improvements  
   - better memory cleanup after http request processing has finished
   - replacing some string concatenations with stringBuffers
   - replacing BufferedInputStreams with serverByteBuffer
   - replacing vectors with arraylists wherever possible
   - replacing hashtables with hashmaps wherever possible
   This was done because function calls to verctor or hashtable functions
   take 3 time longer than calls to functions of arraylists or hashmaps.
   TODO: we should take a look on the class serverObject which is inherited from hashmap
         Do we realy need a synchronization for this class?
   TODO: replace arraylists with linkedLists if random access to the list elements is not needed

*) Robots Parser supports if-modified-since downloads now
   If the downloaded robots.txt file is older than 7 days the robots parser tries to
   download the robots.txt with the if-modified-since header to avoid unnecessary downloads
   if the file was not changed. Additionally the ETag header is used to detect changes.

*) Crawler: better handling of unsupported mimeTypes + FileExtension

*) Bugfix: plasmaWordIndexEntity was not closed correctly in 
   - query.java
   - plasmaswitchboard.java

*) function minimizeUrlDB added to yacy.java 
   this function tests the current urlHashDB for unused urls
   ATTENTION: please don't use this function at the moment because
              it causes the wordIndexDB to flush all words into the
              word directory!

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@853 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2005-10-05 10:45:33 +00:00
orbiter
01db66dc69 implemented image-servlets. the imagetest will stay there only for a limited time. Now images can be generated on-the-fly from servlets 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@852 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2005-10-05 08:40:20 +00:00
orbiter
f6a0e0f162 small bugfix to readFully 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@851 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2005-10-05 07:21:26 +00:00
orbiter
6d5d0ac801 bugfix for startup problems 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@850 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2005-10-05 00:52:55 +00:00
orbiter
5dc0d41900 bugfix in kelondroRA (hint by Martin) 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@847 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2005-10-04 10:24:09 +00:00
orbiter
0c3a20d44f more + changed log for better understanding of outOfMemory bug and others 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@846 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2005-10-04 00:28:59 +00:00
orbiter
e5f8163203 fixed a bug with news; news moving could lead to shurtcut loop / 100% CPU; appeared when clicked on a 'Profile' news in Network menu 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@845 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2005-10-03 23:33:10 +00:00
orbiter
fbb5e36b80 documentation update 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@843 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2005-10-03 23:04:11 +00:00
allo
cd77078aa0 old Version restored before Release 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@842 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2005-10-03 18:10:05 +00:00
allo
a4b747fe97 ProxyAccounts based on userDB 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@841 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2005-10-03 14:26:08 +00:00
theli
0fd9aa6c6e *) Bugfix: supportedFileExt Function didn't detect the file extension correctly because of missing conversion to lower case 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@837 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2005-10-03 10:48:41 +00:00
theli
8a33c9b309 *) Bugfix: supportedFileExt Function didn't detect the file extension correctly if there was a dot 
in one of the parent directories of the file.

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@836 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2005-10-03 10:21:13 +00:00
theli
28c5687ff9 *) Bugfix for "download of non supported file content" via crawler 
See: http://www.yacy-forum.de/viewtopic.php?p=10724#10724

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@835 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2005-10-03 08:45:39 +00:00
theli
2b3f964037 *) Bugfix: supportedFileExt Function didn't chop http parameters before trying to detect the file extension 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@834 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2005-10-03 08:42:55 +00:00
theli
d388292f24 *) adding function for user accounting which is called after each http request 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@827 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2005-09-30 16:02:58 +00:00
theli
c7b7c69484 *) adding some functions to userDB.Entry 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@826 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2005-09-30 15:41:01 +00:00
theli
d9c08a47f9 *) Minor changes to userDB 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@824 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2005-09-30 15:02:17 +00:00
theli
af7829df25 *) adding some functions to userDB.Entry 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@823 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2005-09-30 14:56:50 +00:00
allo
ff1d3d0680 Init of userDB 
Pagelayout of User_p.html


git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@822 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2005-09-30 13:48:26 +00:00
theli
19648702cc *) adding iterator to userDB 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@819 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2005-09-30 13:21:31 +00:00
theli
dd59c1ace9 *) Adding first version of yacy user DB 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@817 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2005-09-30 10:28:49 +00:00
theli
595e0c7e56 *) Bugfix for ProxyErrormsg: Wrong base URL 
See: http://www.yacy-forum.de/viewtopic.php?p=9905#9905

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@815 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2005-09-30 06:15:22 +00:00
allo
9e1485c13b new Class for UserAccounts 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@813 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2005-09-29 20:15:31 +00:00
orbiter
9c4306e41e fixed problem with htcache path 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@811 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2005-09-29 00:24:09 +00:00
orbiter
25a59a51ad fixed problem created with last svn commit 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@810 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2005-09-28 23:16:00 +00:00
orbiter
0054d3b1a6 added age in network menu 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@809 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2005-09-28 22:54:31 +00:00
orbiter
3dd7e90cdd kbytes instead of bytes in performance settings; new default values 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@808 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2005-09-28 18:53:41 +00:00
orbiter
1669eaaa1a fixed svn 805 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@807 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2005-09-28 14:47:57 +00:00
borg-0300
ca82d690a9 changed in SVN 805 one line too much 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@806 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2005-09-28 13:58:42 +00:00
borg-0300
4bb1f849a0 Bugfix for http://www.yacy-forum.de/viewtopic.php?t=1233 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@805 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2005-09-28 13:49:57 +00:00
orbiter
2c7b490e30 memory-logging 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@804 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2005-09-28 00:52:54 +00:00
orbiter
7fc822a59b changed handling of time-zones 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@801 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2005-09-27 16:28:55 +00:00
theli
9b7f37fc37 *) Minor changes 
- more debugging output: storageTime for indexed document is logged now
   - saving memory in plasmaParserDocument.java, plasmaWordIndexEntryContainer.java (not a big deal)

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@798 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2005-09-27 07:10:24 +00:00
orbiter
3c1d968d29 fix-fix for 792 and small changes in ftpc/download/dir experiments 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@797 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2005-09-26 10:36:42 +00:00
theli
b5a8992d29 *) Setting some object fields to final 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@796 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2005-09-26 09:39:54 +00:00
theli
5f95a1cf62 *) Bugfix for ProxyErrormsg: Wrong http host header 
See: http://www.yacy-forum.de/viewtopic.php?p=9905#9905

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@795 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2005-09-26 08:10:40 +00:00
theli
023be89586 *) Bugfix for "Robots.txt wird immer wieder geladen" 
See: http://www.yacy-forum.de/viewtopic.php?p=10241#10233

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@794 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2005-09-26 08:05:59 +00:00
theli
35c6c5ead7 *) Bugfix for "Blacklist und Crawlen" Bug. 
: Crawling continues even if URL is listed in Blacklist
   See: http://www.yacy-forum.de/viewtopic.php?p=10279#10279
   - missing return statement added. Thanks to allo for the
     code review.

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@793 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2005-09-26 06:51:11 +00:00
orbiter
dc474aa22f various bug-fixes 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@792 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2005-09-26 01:10:41 +00:00
orbiter
9e2fc7e5fe load balancing of crawl target domains 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@791 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2005-09-25 01:09:21 +00:00
orbiter
3fcc95a82c integrated crawl-profiles db in memory-performance monitor 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@788 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2005-09-24 00:33:27 +00:00
orbiter
72ce36baba cleanup in kelondroRecords 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@787 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2005-09-24 00:14:01 +00:00
theli
fe6a6abc0b *) Adding robots.txt db to Performance Settings for Memory menue 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@785 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2005-09-23 01:31:29 +00:00
orbiter
3274ae725e increased cache size of robots database; however, this should be integrated into new memory control 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@784 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2005-09-23 00:37:31 +00:00
orbiter
c6d2f50375 changed order of robots and double-check 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@783 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2005-09-23 00:18:08 +00:00
orbiter
68d5ff2ef1 added stringbuffer in condenser 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@782 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2005-09-22 23:43:45 +00:00
theli
90f02ea455 *) removing metainfo from serverargs 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@780 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2005-09-22 23:05:34 +00:00