Marown/yacy_search_server
1
0
Fork 0
mirror of https://github.com/yacy/yacy_search_server.git synced 2024-09-19 00:01:41 +02:00
Code Issues Actions Packages Projects Releases Wiki Activity
4,986 Commits 1 Branch 20 Tags 836 MiB
baae3d91b1
Commit Graph

4986 Commits

This Branch
This Branch
All Branches
Author SHA1 Message Date
low012
baae3d91b1 *) fixed warning when compiling listManager 
*) fixed display of values of information for which part of YaCy (crawler, proxy, ...) blacklist is activated for
*) replaced regular put() with putXML() in several cases

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5305 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-10-27 16:56:19 +00:00
low012
444575e33d *) prevent XSS when importing blacklist 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5304 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-10-27 11:06:38 +00:00
danielr
a4fb76e93c undo r5300 (not fixed as seen after longer run) 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5303 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-10-25 23:20:09 +00:00
low012
a99a629ed4 *) quick fix to prevent comments for blog entries which don't exist (http://forum.yacy-websuche.de/viewtopic.php?f=5&t=1554) 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5302 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-10-25 12:04:10 +00:00
low012
00e27e5050 *) fixed bug which made it possible to write files outside of the DATA/LIST directory when creating a new blacklist 
*) a blacklist will only be created if no blacklist with same name exists (some refactoring has been necessary for this)
*) further minor fixes
*) to be continued...

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5301 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-10-25 00:11:03 +00:00
danielr
0f9c0bd0d5 fix for ConcurrentModificationException at de.anomic.index.indexContainerHeap$heapCacheIterator.next(indexContainerHeap.java:324) 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5300 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-10-24 14:00:41 +00:00
danielr
103ad2a437 some javadoc 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5299 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-10-24 13:58:26 +00:00
orbiter
b098522977 some very small advances to index utf-8 (not working yet), inserted also debugging code 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5298 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-10-22 22:04:13 +00:00
orbiter
2f49666908 integrated the character decoding into the parser, removed old code 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5297 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-10-22 20:56:13 +00:00
orbiter
49293c1358 fix for deadlock in new encoder :-( 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5296 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-10-22 19:36:34 +00:00
orbiter
0edec2b760 FULL redesign of algorithms in htmlTools to encode/decode strings from/to unicode and html. 
The old process used a not really efficient way to detect html encoding strings in texts.
All calling methods had been adoped to call the new class in an enhanced way with less parameters.

Many classes in interfaces used a XML encoding only (instead of full html conversion from unicode to html); this behavior was not changed with this commit but should be controlled again since it points out possible XSS leaks

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5295 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-10-22 18:59:04 +00:00
orbiter
958ec20cd0 removed specialized umlaute-handling in html parser. This has to be replaced by something that is able to transfer all possible html encodings into utf-8. Please see SVN 5293 for test cases. 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5294 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-10-22 11:11:55 +00:00
orbiter
204220ecd5 added test files for UTF-8 / Umlaute - Testing: 
These 3 files contain the same text in different HTML encodings. We use this documents to test if the parser and indexer creates the same set of word hashes for all three texts.

To use these files, run a indexing/crawling on them. To get the files inside the localhost-path, do the following:

cd <yacy-home>
rmdir DATA/HTDOCS/repository
ln -s test/parsertest DATA/HTDOCS/repository

you have then linked the test directory as repository directory which you can reach in yacy if you switch to intranet indexing mode. So the next step is to start yacy, then
- switch to intranet use case
- go to the crawl start page
- the repository directory should be the default path as crawl start
- start the crawl
- search for any word that appears in the demo texts
- search not only for words with umlautel but also for words without umlaute to ensure that you find _all_ three documents
- see how yacy presents the snippet with the text containing umlaute

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5293 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-10-22 11:07:14 +00:00
f1ori
2e53cbc66a should compile now 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5292 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-10-22 08:50:30 +00:00
f1ori
f3bf2e379e should compile again 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5291 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-10-22 07:35:49 +00:00
f1ori
dd8441f102 fix bug: data from plasmaParser is allready converted to UTF-8 
After removing the restrictions in the code, YaCy should be able to index Unicode-charaters!


git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5290 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-10-21 20:19:10 +00:00
orbiter
47f0c3b002 replaced the cacheAdmin with the ViewFile servlet, because the cacheAdmin was an interface to the old HTCACHE data structure which does not exist any more. Changed links to point to the ViewFile servlets. 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5289 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-10-21 11:27:50 +00:00
orbiter
6941bf42b1 performance hacks 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5288 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-10-20 14:07:09 +00:00
orbiter
9b0c4b1063 redesign of parts of the new BLOB buffer 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5287 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-10-19 22:30:44 +00:00
orbiter
1778fb420d - added some performance tweaks to the new BLOB buffer 
- removed the now superfluous HT storage thread
- reduced number of file decompression by shifting the compression moment to the future


git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5286 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-10-19 18:10:42 +00:00
low012
77e41da7d2 *) further propagation of display value (see http://forum.yacy-websuche.de/viewtopic.php?f=5&t=1536) 
*) removed another depreciated parameter "time" which led to ugly -UNRESOLVED_PATTERN- in URL

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5285 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-10-18 19:39:46 +00:00
orbiter
9663e61449 added another class to handle BLOB writings to the new HTCACHE data storage: 
- entries are buffered and written as stream with many entries at once (saves many IO accesses)
- entries are compressed with gzip: increases capacity of cache
- concurrency for stream-writing and compression: all writings to the cache are non-blocking

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5284 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-10-18 08:57:48 +00:00
low012
ff46ce8520 *) fixed display=2 (see http://forum.yacy-websuche.de/viewtopic.php?f=5&t=1536) 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5283 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-10-17 19:57:39 +00:00
orbiter
382226da94 fix for bug introduced in SVN 5281: parameters were switched 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5282 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-10-17 12:14:57 +00:00
danielr
f2fd043797 refactoring (moved duplicate code into methods) 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5281 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-10-17 10:39:32 +00:00
danielr
c612046e5e r5278 java 1.5 compatible 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5280 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-10-17 09:59:59 +00:00
f1ori
af71ec93bf ops, forgot to import something 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5279 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-10-16 22:44:25 +00:00
f1ori
9e65e9141c * always use UTF-8 for encoding hashes 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5278 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-10-16 22:35:27 +00:00
f1ori
03d2b323c2 * remove bad mimetype tag so I see my changes 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5277 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-10-16 22:31:38 +00:00
orbiter
826ca79735 refactoring and new architecture to store the files of the web cache: 
- files are not stored any more as individual files
- a new database structure using BLOBHeap files stores many cache entries in common files
- all file-writing procedures had been migrated to generate byte[] objects which are written with the new database methods

this is only an intermediate step to the final architecture, where cached files are written together with their metadata in one single database structure.

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5276 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-10-16 21:24:09 +00:00
low012
2b18a9b2c4 *) removed depreciated parameter "time" which led to ugly -UNRESOLVED_PATTERN- in URL 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5275 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-10-16 19:31:29 +00:00
danielr
f095137238 - respecting httpdMaxBusySessions (refusing new connections if limit is hit) 
- comments in serverBusyThread converted to JavaDoc
- better debug output for npe-case in diskUsage


git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5274 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-10-16 10:53:32 +00:00
danielr
94e43ece41 unter debian yacy als (fast) letztes starten und erstes beenden 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5271 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-10-14 21:53:25 +00:00
f1ori
9d69964d3d start daemon in different runlevels on debian and fedora/openSUSE-Systems 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5270 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-10-14 15:28:24 +00:00
orbiter
8ba33f104e fix for npe 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5269 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-10-13 21:59:53 +00:00
orbiter
998861acfd - some refactoring in BLOBHeap to enable more gap processing functions 
- better gap merging in BLOBHeap
- shrinking of heap file if gap is at end of file when file is closed

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5268 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-10-13 21:15:54 +00:00
lotus
9d50bfd0b3 fix for npe: http://forum.yacy-websuche.de/viewtopic.php?p=10562 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5267 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-10-13 09:09:53 +00:00
orbiter
766cad6e93 enhancement in memory management of BLOB Heap files / merging of deleted entries 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5266 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-10-12 22:15:01 +00:00
orbiter
7860d5d632 fix for bug in seed list management (cause was bad class overloading, only visual effects!) 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5265 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-10-12 19:51:53 +00:00
lotus
603282bcf4 fix for out of bounds exception 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5264 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-10-11 07:47:34 +00:00
orbiter
ffed5fc415 fixed problem with lost peers in database 
migrated seedDB from BLOBTree to BLOBHeap

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5263 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-10-10 14:40:02 +00:00
orbiter
6fb865fbdc - fix of bug in iterator in kelondroBLOBHeap which caused bug in crawl profile listing 
- some refactoring of classes that use kelondroMap (Map instead of HashMap)

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5262 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-10-10 08:39:11 +00:00
orbiter
2d65887723 - fix for bug in new profile handling 
- added a new feature in ymageChart (cannot be seen yet, just wait... will be used in profiling chart)

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5261 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-10-09 22:31:43 +00:00
lotus
4df63626f5 sorry 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5260 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-10-09 13:51:43 +00:00
lotus
736dd86193 - option enableSimpleConfig can disable hidden tables 
- corrected some Xmx values
- friendlier welcome message format

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5259 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-10-09 12:48:43 +00:00
orbiter
ff68f394dd fix for problem with balancer and lost crawl profiles: 
if crawl profile ist lost, no robots.txt is loaded any more

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5258 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-10-08 18:26:36 +00:00
f1ori
d197a62faf * wrong default parameter in initscript 
* should fix http://forum.yacy-websuche.de/viewtopic.php?f=5&t=1500


git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5257 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-10-08 10:54:25 +00:00
danielr
f0c7166a48 fix for http://forum.yacy-websuche.de/viewtopic.php?f=6&t=1505 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5256 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-10-08 09:48:04 +00:00
f1ori
88fdd4c4f2 * remove unix wrappers from installonlinux-target, because they are useless on systemwide installations 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5255 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-10-05 22:13:09 +00:00
daburna
9286afad4e new development cycle - nun aber wirklich :-P 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5254 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-10-05 22:08:14 +00:00