Marown/yacy_search_server
1
0
Fork 0
mirror of https://github.com/yacy/yacy_search_server.git synced 2024-09-21 00:00:13 +02:00
Code Issues Actions Packages Projects Releases Wiki Activity
858 Commits 1 Branch 20 Tags 836 MiB
2851658c2a
Commit Graph

551 Commits

Author SHA1 Message Date
orbiter
2851658c2a re-integrated Martins last change to crawl stacker from svn 882 that I had deleted accidently 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@888 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2005-10-09 16:11:41 +00:00
orbiter
c83594528c integrated crawl stacker into thread control 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@887 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2005-10-09 15:59:09 +00:00
orbiter
16a49c1c9d fix for graphics generation bug, see http://www.yacy-forum.de/viewtopic.php?p=10987#10987 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@886 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2005-10-09 14:46:33 +00:00
orbiter
d666b61b83 fix for news-deletion, see also http://www.yacy-forum.de/viewtopic.php?p=11000#11000 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@885 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2005-10-09 14:31:35 +00:00
borg-0300
4180c422e8 cleaned, finals, Properties 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@884 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2005-10-09 13:18:48 +00:00
theli
959eefbc4f *) Robots.txt parser/ppt 
cutting of comments at the line end
*) Adding Threadpool for stackCrawl Thread to speedup robots.txt download
   and double url checks

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@882 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2005-10-09 04:43:07 +00:00
orbiter
c1c94111b0 added new network picture at Network menu using the new image-servlet method 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@880 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2005-10-08 00:37:43 +00:00
orbiter
d98ed41f31 update to image painter and test on http://localhost:8080/imagetest.png 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@878 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2005-10-07 14:17:33 +00:00
allo
f65c939a60 userDB Auth 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@874 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2005-10-07 13:49:07 +00:00
orbiter
5153ec0f3e update to image painter 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@873 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2005-10-07 01:25:39 +00:00
orbiter
1b2db0b52a fix for file-share access; damaged some commits before by me :-( 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@870 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2005-10-06 22:30:13 +00:00
orbiter
1a5d98cd6d better imagePainter example and fix for typo http://www.yacy-forum.de/viewtopic.php?p=10920#10920 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@868 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2005-10-06 11:51:35 +00:00
borg-0300
afc5ef2819 cleaned, Properties 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@867 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2005-10-06 09:41:59 +00:00
orbiter
60ab32c313 removed obsolete htmlPlotter 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@865 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2005-10-05 16:47:29 +00:00
orbiter
06f4ac494a image-painter test; try with http://localhost:8080/imagetest.png 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@864 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2005-10-05 16:38:16 +00:00
borg-0300
fb27428674 added restart to Status.html 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@863 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2005-10-05 16:35:05 +00:00
borg-0300
0dcfc25d98 cleaned, finals, Properties 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@862 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2005-10-05 16:25:43 +00:00
borg-0300
4191b21e73 cleaned, finals, Properties 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@858 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2005-10-05 13:30:12 +00:00
orbiter
f6cf3967de fix for compile-bug in svn 583 (Martin guck mal ob das richtig ist: fifo oder filo-stack?) 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@854 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2005-10-05 12:21:30 +00:00
theli
a2fa75e688 *) Asynchronous queuing of crawl job URLs (stackCrawl) 
various checks like the blacklist check or the robots.txt disallow check are now
   done by a separate thread to unburden the indexer thread(s)
   TODO: maybe we have to introduce a threadpool here if it turn out that this single
         thread is a bottleneck because of the time consuming robots.txt downloads

*) improved index transfer
   The index selection and transmission is done in parallel now to improve index 
   transfer performance.
   TODO: maybe we could speed up performance by unsing multiple transmission threads in 
         parallel instead of only a single one.

*) gzip encoded post requests
   it is now configureable if a gzip encoded post request should be send on
   intex transfer/distribution

*) storage Peer (very experimentell and not optimized yet)
   Now it's possible to send the result of the yacy indexer thread to a remote peer 
   istead of storing the indexed words locally. 
   This could be done by setting the property "storagePeerHash" in the yacy config file
   - Please note that if the index transfer fails, the index ist stored locally.
   - TODO: currently this index transfer is done by the indexer thread. 
     To seedup the indexer
     a) this transmission should be done in parallel and
     b) multiple chunks should be bundled and transfered together


*) general performance improvements  
   - better memory cleanup after http request processing has finished
   - replacing some string concatenations with stringBuffers
   - replacing BufferedInputStreams with serverByteBuffer
   - replacing vectors with arraylists wherever possible
   - replacing hashtables with hashmaps wherever possible
   This was done because function calls to verctor or hashtable functions
   take 3 time longer than calls to functions of arraylists or hashmaps.
   TODO: we should take a look on the class serverObject which is inherited from hashmap
         Do we realy need a synchronization for this class?
   TODO: replace arraylists with linkedLists if random access to the list elements is not needed

*) Robots Parser supports if-modified-since downloads now
   If the downloaded robots.txt file is older than 7 days the robots parser tries to
   download the robots.txt with the if-modified-since header to avoid unnecessary downloads
   if the file was not changed. Additionally the ETag header is used to detect changes.

*) Crawler: better handling of unsupported mimeTypes + FileExtension

*) Bugfix: plasmaWordIndexEntity was not closed correctly in 
   - query.java
   - plasmaswitchboard.java

*) function minimizeUrlDB added to yacy.java 
   this function tests the current urlHashDB for unused urls
   ATTENTION: please don't use this function at the moment because
              it causes the wordIndexDB to flush all words into the
              word directory!

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@853 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2005-10-05 10:45:33 +00:00
orbiter
01db66dc69 implemented image-servlets. the imagetest will stay there only for a limited time. Now images can be generated on-the-fly from servlets 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@852 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2005-10-05 08:40:20 +00:00
orbiter
f6a0e0f162 small bugfix to readFully 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@851 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2005-10-05 07:21:26 +00:00
orbiter
6d5d0ac801 bugfix for startup problems 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@850 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2005-10-05 00:52:55 +00:00
orbiter
5dc0d41900 bugfix in kelondroRA (hint by Martin) 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@847 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2005-10-04 10:24:09 +00:00
orbiter
0c3a20d44f more + changed log for better understanding of outOfMemory bug and others 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@846 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2005-10-04 00:28:59 +00:00
orbiter
e5f8163203 fixed a bug with news; news moving could lead to shurtcut loop / 100% CPU; appeared when clicked on a 'Profile' news in Network menu 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@845 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2005-10-03 23:33:10 +00:00
orbiter
fbb5e36b80 documentation update 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@843 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2005-10-03 23:04:11 +00:00
allo
cd77078aa0 old Version restored before Release 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@842 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2005-10-03 18:10:05 +00:00
allo
a4b747fe97 ProxyAccounts based on userDB 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@841 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2005-10-03 14:26:08 +00:00
theli
0fd9aa6c6e *) Bugfix: supportedFileExt Function didn't detect the file extension correctly because of missing conversion to lower case 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@837 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2005-10-03 10:48:41 +00:00
theli
8a33c9b309 *) Bugfix: supportedFileExt Function didn't detect the file extension correctly if there was a dot 
in one of the parent directories of the file.

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@836 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2005-10-03 10:21:13 +00:00
theli
28c5687ff9 *) Bugfix for "download of non supported file content" via crawler 
See: http://www.yacy-forum.de/viewtopic.php?p=10724#10724

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@835 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2005-10-03 08:45:39 +00:00
theli
2b3f964037 *) Bugfix: supportedFileExt Function didn't chop http parameters before trying to detect the file extension 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@834 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2005-10-03 08:42:55 +00:00
theli
d388292f24 *) adding function for user accounting which is called after each http request 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@827 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2005-09-30 16:02:58 +00:00
theli
c7b7c69484 *) adding some functions to userDB.Entry 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@826 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2005-09-30 15:41:01 +00:00
theli
d9c08a47f9 *) Minor changes to userDB 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@824 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2005-09-30 15:02:17 +00:00
theli
af7829df25 *) adding some functions to userDB.Entry 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@823 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2005-09-30 14:56:50 +00:00
allo
ff1d3d0680 Init of userDB 
Pagelayout of User_p.html


git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@822 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2005-09-30 13:48:26 +00:00
theli
19648702cc *) adding iterator to userDB 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@819 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2005-09-30 13:21:31 +00:00
theli
dd59c1ace9 *) Adding first version of yacy user DB 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@817 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2005-09-30 10:28:49 +00:00
theli
595e0c7e56 *) Bugfix for ProxyErrormsg: Wrong base URL 
See: http://www.yacy-forum.de/viewtopic.php?p=9905#9905

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@815 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2005-09-30 06:15:22 +00:00
allo
9e1485c13b new Class for UserAccounts 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@813 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2005-09-29 20:15:31 +00:00
orbiter
9c4306e41e fixed problem with htcache path 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@811 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2005-09-29 00:24:09 +00:00
orbiter
25a59a51ad fixed problem created with last svn commit 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@810 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2005-09-28 23:16:00 +00:00
orbiter
0054d3b1a6 added age in network menu 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@809 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2005-09-28 22:54:31 +00:00
orbiter
3dd7e90cdd kbytes instead of bytes in performance settings; new default values 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@808 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2005-09-28 18:53:41 +00:00
orbiter
1669eaaa1a fixed svn 805 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@807 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2005-09-28 14:47:57 +00:00
borg-0300
ca82d690a9 changed in SVN 805 one line too much 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@806 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2005-09-28 13:58:42 +00:00
borg-0300
4bb1f849a0 Bugfix for http://www.yacy-forum.de/viewtopic.php?t=1233 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@805 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2005-09-28 13:49:57 +00:00
orbiter
2c7b490e30 memory-logging 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@804 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2005-09-28 00:52:54 +00:00