Marown/yacy_search_server
1
0
Fork 0
mirror of https://github.com/yacy/yacy_search_server.git synced 2024-09-21 00:00:13 +02:00
Code Issues Actions Packages Projects Releases Wiki Activity
836 Commits 1 Branch 20 Tags 836 MiB
06f4ac494a
Commit Graph

345 Commits

Author SHA1 Message Date
orbiter
06f4ac494a image-painter test; try with http://localhost:8080/imagetest.png 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@864 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2005-10-05 16:38:16 +00:00
borg-0300
fb27428674 added restart to Status.html 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@863 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2005-10-05 16:35:05 +00:00
borg-0300
13dd16d1dd small change 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@861 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2005-10-05 16:00:16 +00:00
borg-0300
5c4db6968e sorted URL list, StringBuffers, finals, cleaned, Properties 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@860 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2005-10-05 14:11:50 +00:00
borg-0300
2a18e6486b cleaned, finals, Properties 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@859 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2005-10-05 13:45:11 +00:00
borg-0300
64acb46a91 cleaned, finals, Properties 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@857 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2005-10-05 13:16:53 +00:00
borg-0300
52168fab9b cleaned, finals, Properties 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@856 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2005-10-05 13:14:18 +00:00
orbiter
be60b512b7 fix for last commit 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@855 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2005-10-05 12:24:25 +00:00
orbiter
f6cf3967de fix for compile-bug in svn 583 (Martin guck mal ob das richtig ist: fifo oder filo-stack?) 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@854 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2005-10-05 12:21:30 +00:00
theli
a2fa75e688 *) Asynchronous queuing of crawl job URLs (stackCrawl) 
various checks like the blacklist check or the robots.txt disallow check are now
   done by a separate thread to unburden the indexer thread(s)
   TODO: maybe we have to introduce a threadpool here if it turn out that this single
         thread is a bottleneck because of the time consuming robots.txt downloads

*) improved index transfer
   The index selection and transmission is done in parallel now to improve index 
   transfer performance.
   TODO: maybe we could speed up performance by unsing multiple transmission threads in 
         parallel instead of only a single one.

*) gzip encoded post requests
   it is now configureable if a gzip encoded post request should be send on
   intex transfer/distribution

*) storage Peer (very experimentell and not optimized yet)
   Now it's possible to send the result of the yacy indexer thread to a remote peer 
   istead of storing the indexed words locally. 
   This could be done by setting the property "storagePeerHash" in the yacy config file
   - Please note that if the index transfer fails, the index ist stored locally.
   - TODO: currently this index transfer is done by the indexer thread. 
     To seedup the indexer
     a) this transmission should be done in parallel and
     b) multiple chunks should be bundled and transfered together


*) general performance improvements  
   - better memory cleanup after http request processing has finished
   - replacing some string concatenations with stringBuffers
   - replacing BufferedInputStreams with serverByteBuffer
   - replacing vectors with arraylists wherever possible
   - replacing hashtables with hashmaps wherever possible
   This was done because function calls to verctor or hashtable functions
   take 3 time longer than calls to functions of arraylists or hashmaps.
   TODO: we should take a look on the class serverObject which is inherited from hashmap
         Do we realy need a synchronization for this class?
   TODO: replace arraylists with linkedLists if random access to the list elements is not needed

*) Robots Parser supports if-modified-since downloads now
   If the downloaded robots.txt file is older than 7 days the robots parser tries to
   download the robots.txt with the if-modified-since header to avoid unnecessary downloads
   if the file was not changed. Additionally the ETag header is used to detect changes.

*) Crawler: better handling of unsupported mimeTypes + FileExtension

*) Bugfix: plasmaWordIndexEntity was not closed correctly in 
   - query.java
   - plasmaswitchboard.java

*) function minimizeUrlDB added to yacy.java 
   this function tests the current urlHashDB for unused urls
   ATTENTION: please don't use this function at the moment because
              it causes the wordIndexDB to flush all words into the
              word directory!

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@853 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2005-10-05 10:45:33 +00:00
orbiter
01db66dc69 implemented image-servlets. the imagetest will stay there only for a limited time. Now images can be generated on-the-fly from servlets 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@852 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2005-10-05 08:40:20 +00:00
borg-0300
a9c466ef21 cleaned, finals, StringBuffer, Properties 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@849 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2005-10-04 17:51:32 +00:00
orbiter
0c3a20d44f more + changed log for better understanding of outOfMemory bug and others 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@846 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2005-10-04 00:28:59 +00:00
borg-0300
d8544e8416 cosmetic 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@844 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2005-10-03 23:13:41 +00:00
allo
0cb9566546 HTML for last Commit. 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@840 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2005-10-03 12:23:20 +00:00
allo
8d871a6a79 delete Function 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@839 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2005-10-03 12:17:12 +00:00
allo
7df4c85506 bugfix & cleanup 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@838 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2005-10-03 11:51:12 +00:00
borg-0300
f58cfbf9a8 passive red to yellow changed 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@833 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2005-10-01 14:24:59 +00:00
allo
252492d031 update User 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@832 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2005-10-01 13:34:13 +00:00
borg-0300
1b61b674db changed method 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@831 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2005-10-01 11:43:48 +00:00
borg-0300
2d481e89df Michael would like it 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@829 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2005-09-30 16:06:42 +00:00
allo
60fc32ffeb load values 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@828 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2005-09-30 16:06:20 +00:00
allo
8de3b2dd3c User create function 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@825 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2005-09-30 15:25:45 +00:00
allo
ff1d3d0680 Init of userDB 
Pagelayout of User_p.html


git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@822 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2005-09-30 13:48:26 +00:00
borg-0300
8d728d742b remove my SVN 818 change 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@821 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2005-09-30 13:46:25 +00:00
borg-0300
a3bb20ca88 New column Info replaced Type, Contact and accept Crawl/Index; 
11 new pictures show the information in 11x32 pixel;


git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@818 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2005-09-30 13:03:36 +00:00
theli
1d1b8f699e *) Extending proxy autoconfig to avoid problems with multiple local network cards 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@816 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2005-09-30 07:47:28 +00:00
low012
394fb5d484 *) Small changes that make entering values much easier for people who use the TAB-key to navigate through the page or who use textbowsers like lynx. 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@814 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2005-09-29 20:17:10 +00:00
borg-0300
138d218d5b all big number have now separators (not Last Seen); 
used dot instead of comma;
used yacySeed.constants;

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@812 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2005-09-29 13:10:59 +00:00
orbiter
9c4306e41e fixed problem with htcache path 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@811 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2005-09-29 00:24:09 +00:00
orbiter
25a59a51ad fixed problem created with last svn commit 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@810 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2005-09-28 23:16:00 +00:00
orbiter
0054d3b1a6 added age in network menu 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@809 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2005-09-28 22:54:31 +00:00
orbiter
3dd7e90cdd kbytes instead of bytes in performance settings; new default values 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@808 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2005-09-28 18:53:41 +00:00
orbiter
2c7b490e30 memory-logging 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@804 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2005-09-28 00:52:54 +00:00
allo
ee347364d9 basic xhtml/css layout 
-to test, replace #[he4ader]# with #[xhtml-header]#, #[footer]# with #[xhtml-footer]# and use the xhtml-style css


git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@803 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2005-09-27 20:13:34 +00:00
orbiter
7fc822a59b changed handling of time-zones 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@801 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2005-09-27 16:28:55 +00:00
orbiter
3c1d968d29 fix-fix for 792 and small changes in ftpc/download/dir experiments 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@797 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2005-09-26 10:36:42 +00:00
orbiter
dc474aa22f various bug-fixes 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@792 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2005-09-26 01:10:41 +00:00
(no author)
1aa79f5bb5 cleaned; 
Properties;

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@790 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2005-09-24 11:58:17 +00:00
orbiter
694226d4b1 typo 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@789 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2005-09-24 00:40:52 +00:00
orbiter
3fcc95a82c integrated crawl-profiles db in memory-performance monitor 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@788 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2005-09-24 00:33:27 +00:00
theli
fe6a6abc0b *) Adding robots.txt db to Performance Settings for Memory menue 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@785 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2005-09-23 01:31:29 +00:00
orbiter
68d5ff2ef1 added stringbuffer in condenser 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@782 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2005-09-22 23:43:45 +00:00
theli
c42a543bc3 *) Adding peername to logmessage when receiving URLs 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@781 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2005-09-22 23:40:03 +00:00
orbiter
495bc8bec6 removed cache-control from low and medium priority caches which reduces memory use and computation overhead 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@774 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2005-09-22 20:01:26 +00:00
theli
1dc94e7753 *) Adding support for gzip content-encoding of http post requests 
used to transferRWIs and transferURLs.
   See: http://www.yacy-forum.de/viewtopic.php?t=1167#10020

*) adding yacyVersion.java containing constants defining yacy versions
   that support a given feature.
   Needed to determine if a remote peer is able to decode gzip 
   content-encoded http post bodies properly.

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@772 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2005-09-22 10:30:55 +00:00
orbiter
e6b9b23290 configuration of startup-memory in webinterface 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@771 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2005-09-22 01:17:49 +00:00
orbiter
07f30931ec various configuration options in memory performance 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@763 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2005-09-21 14:21:45 +00:00
orbiter
2f732e32a2 enhancements to memory menue 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@762 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2005-09-21 12:21:01 +00:00
borg-0300
150bd33591 finals; 
cleaned;
Properties;

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@761 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2005-09-21 10:48:46 +00:00