Marown/yacy_search_server
1
0
Fork 0
mirror of https://github.com/yacy/yacy_search_server.git synced 2024-09-21 00:00:13 +02:00
Code Issues Actions Packages Projects Releases Wiki Activity
1,400 Commits 1 Branch 20 Tags 836 MiB
47344e8df0
Commit Graph

72 Commits

Author SHA1 Message Date
orbiter
a56fefe0d3 added missing forced-flush for index cache 
see http://www.yacy-forum.de/viewtopic.php?p=15732#15732

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1434 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2006-01-24 16:24:15 +00:00
orbiter
eabf4a0386 fix for null pointer exception during shut-down 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1415 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2006-01-23 13:45:14 +00:00
orbiter
90b940e90e fixed position storage problem. 
Now the word position is properly stored.
No use of that now, but can be used for better ranking.

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1378 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2006-01-20 00:26:55 +00:00
orbiter
0371494010 tried to add word position to index 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1377 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2006-01-19 14:13:39 +00:00
hermens
ca7407b7e1 *) Don't change maxTime if zero or negative 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1363 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2006-01-17 14:47:21 +00:00
hermens
971247b78f - rotate merged indexes after merging 
see: http://www.yacy-forum.de/viewtopic.php?t=1717
- fix -rwihashlist to correctly shutdown



git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1336 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2006-01-13 23:59:04 +00:00
orbiter
2028403670 - consolidated different orderings to kelondroNaturalOrder 
- added another iteration method to rwihash-enumeration


git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1309 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2006-01-11 00:32:44 +00:00
orbiter
9544c47684 added some UTF-8 handling. 
hope this will help somehow.. for shure not THE solution to our UTF-8 problem


git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1308 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2006-01-10 16:48:59 +00:00
orbiter
83a34b838d * added Object allocation monitor on performanceMemory page 
* added some final statements
* changed shutdown sequence order

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1211 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2005-12-14 13:04:43 +00:00
orbiter
3031903d50 re-design of RAM cache flush into assortment cluster 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1209 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2005-12-13 16:00:20 +00:00
orbiter
0c762daf4b better startup failure handling 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1205 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2005-12-12 23:59:58 +00:00
orbiter
3d8a5ae652 code cleanup 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1166 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2005-12-05 14:24:13 +00:00
theli
ca26aab9b1 *) More debugging output for migrateWords 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1085 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2005-11-15 11:55:09 +00:00
orbiter
9913049009 fixed outOfMemory bug caused by loops in kelondroTree during enumeration 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1079 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2005-11-15 01:20:05 +00:00
orbiter
02f8013013 auto-delete of corrupted word files during word-migration 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1047 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2005-11-07 14:57:37 +00:00
borg-0300
00ab4d8723 cleaned, small change, Properties 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1026 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2005-11-04 13:41:51 +00:00
hydrox
56b9f34411 *)removed unused imports 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1015 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2005-11-02 16:30:45 +00:00
orbiter
4dcbc26ef1 introduction of search profiles; very experimental 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@976 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2005-10-23 17:50:27 +00:00
orbiter
6260942590 changed search process: received indexes are now buffered and written to wordIndex after search 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@934 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2005-10-13 13:57:15 +00:00
orbiter
10d3627c90 changed word cache flush scheduling and removed possible locks 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@910 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2005-10-11 07:06:33 +00:00
orbiter
839db8869c added high/low priority for index adding 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@899 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2005-10-10 09:28:28 +00:00
theli
959eefbc4f *) Robots.txt parser/ppt 
cutting of comments at the line end
*) Adding Threadpool for stackCrawl Thread to speedup robots.txt download
   and double url checks

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@882 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2005-10-09 04:43:07 +00:00
theli
a2fa75e688 *) Asynchronous queuing of crawl job URLs (stackCrawl) 
various checks like the blacklist check or the robots.txt disallow check are now
   done by a separate thread to unburden the indexer thread(s)
   TODO: maybe we have to introduce a threadpool here if it turn out that this single
         thread is a bottleneck because of the time consuming robots.txt downloads

*) improved index transfer
   The index selection and transmission is done in parallel now to improve index 
   transfer performance.
   TODO: maybe we could speed up performance by unsing multiple transmission threads in 
         parallel instead of only a single one.

*) gzip encoded post requests
   it is now configureable if a gzip encoded post request should be send on
   intex transfer/distribution

*) storage Peer (very experimentell and not optimized yet)
   Now it's possible to send the result of the yacy indexer thread to a remote peer 
   istead of storing the indexed words locally. 
   This could be done by setting the property "storagePeerHash" in the yacy config file
   - Please note that if the index transfer fails, the index ist stored locally.
   - TODO: currently this index transfer is done by the indexer thread. 
     To seedup the indexer
     a) this transmission should be done in parallel and
     b) multiple chunks should be bundled and transfered together


*) general performance improvements  
   - better memory cleanup after http request processing has finished
   - replacing some string concatenations with stringBuffers
   - replacing BufferedInputStreams with serverByteBuffer
   - replacing vectors with arraylists wherever possible
   - replacing hashtables with hashmaps wherever possible
   This was done because function calls to verctor or hashtable functions
   take 3 time longer than calls to functions of arraylists or hashmaps.
   TODO: we should take a look on the class serverObject which is inherited from hashmap
         Do we realy need a synchronization for this class?
   TODO: replace arraylists with linkedLists if random access to the list elements is not needed

*) Robots Parser supports if-modified-since downloads now
   If the downloaded robots.txt file is older than 7 days the robots parser tries to
   download the robots.txt with the if-modified-since header to avoid unnecessary downloads
   if the file was not changed. Additionally the ETag header is used to detect changes.

*) Crawler: better handling of unsupported mimeTypes + FileExtension

*) Bugfix: plasmaWordIndexEntity was not closed correctly in 
   - query.java
   - plasmaswitchboard.java

*) function minimizeUrlDB added to yacy.java 
   this function tests the current urlHashDB for unused urls
   ATTENTION: please don't use this function at the moment because
              it causes the wordIndexDB to flush all words into the
              word directory!

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@853 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2005-10-05 10:45:33 +00:00
theli
b5a8992d29 *) Setting some object fields to final 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@796 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2005-09-26 09:39:54 +00:00
orbiter
495bc8bec6 removed cache-control from low and medium priority caches which reduces memory use and computation overhead 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@774 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2005-09-22 20:01:26 +00:00
orbiter
fb52a82008 added new performance page for memory settings 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@751 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2005-09-20 10:10:34 +00:00
rramthun
4036ee812a Updated german language file 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@721 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2005-09-13 16:29:59 +00:00
orbiter
43b42854a0 fix for null-entries and http://www.yacy-forum.de/viewtopic.php?p=8649 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@699 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2005-09-11 03:54:52 +00:00
theli
4fd5b95b1f *) Renaming Logger function names to reflect the proper Java Logging API Loglevels 
- please use logFine instead of logDebug
   - please use logSevere instead of logFailure and logError
   See: http://www.yacy-forum.de/viewtopic.php?p=8726#8726

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@615 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2005-08-30 21:32:59 +00:00
theli
6adf8a4bde *) Renaming Logger function names to reflect the proper Java Logging API Loglevels 
- please use logFine instead of logDebug
   - please use logFailure instead of logError
   See: http://www.yacy-forum.de/viewtopic.php?p=8726#8726

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@614 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2005-08-30 21:10:39 +00:00
theli
b33094e925 *) Trying to solve "Too many open files bug" 
*) Temp.Bugfix for "Bug in Index Restore"
   See: http://www.yacy-forum.de/viewtopic.php?p=8647#8647
   Orbiter: Please take a look



git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@602 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2005-08-30 09:07:42 +00:00
orbiter
ba0a486328 moved printStackTrace() to logging 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@539 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2005-08-14 23:35:18 +00:00
orbiter
3094045d34 fix for http://www.yacy-forum.de/viewtopic.php?p=7454#7454 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@536 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2005-08-14 16:16:11 +00:00
orbiter
cd10370992 several bugfixes and dht selection / logging improvement 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@531 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2005-08-14 00:57:30 +00:00
orbiter
5716f8521d bug fixes for word ordering and dht index selection 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@521 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2005-08-12 14:06:47 +00:00
orbiter
f5259f29e8 word cache behaviour fix and other fixes 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@519 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2005-08-11 23:33:19 +00:00
orbiter
248c24b60a intermission-feature usage in case of local and remote search 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@510 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2005-08-09 20:43:37 +00:00
orbiter
2d8557cb10 minor changes 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@487 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2005-08-03 02:02:39 +00:00
orbiter
b73557ed2d better assortment monitoring and enhanced profile menue 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@416 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2005-07-20 13:03:41 +00:00
orbiter
1f36bf4dae enhanced assortment capacity; added extended WORDS migration 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@412 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2005-07-20 00:39:06 +00:00
orbiter
9c72b4cdec replaced index dump stack by an dump array and limited url number in assortment ram (prevents too much RAM occupation) 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@406 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2005-07-18 13:32:44 +00:00
orbiter
159f795f65 bugfix (null pointer exception in assortments) 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@404 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2005-07-17 22:25:50 +00:00
orbiter
1d2155675b changed assortment memory cache flush 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@403 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2005-07-17 21:22:18 +00:00
orbiter
2f0d7ea8d3 removed htcache stati (superfluous now) 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@396 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2005-07-09 00:33:34 +00:00
orbiter
858cd94299 replaced indexing ram-queue by file-based stack-queue 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@381 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2005-07-06 14:48:41 +00:00
orbiter
d6c85228a6 enhanced snippet computation 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@319 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2005-06-23 12:12:12 +00:00
orbiter
1e7f062350 many bugfixes, memory leak fixes, performance enhancements; new kelondroHashtable; activated snippets 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@313 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2005-06-23 02:07:45 +00:00
orbiter
68dc2b0c6b added kelondroArray, the basis for upcoming kelondroHash and some bug fixes 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@311 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2005-06-21 01:17:25 +00:00
orbiter
e3c92818db avoiding OutOfMemoryError routines 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@302 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2005-06-19 13:37:17 +00:00
orbiter
a25b5b4986 fixed possible memory leak in htmlScraper: be aware that now links can get lost; further work necessary 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@288 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2005-06-16 18:31:28 +00:00