Marown/yacy_search_server
1
0
Fork 0
mirror of https://github.com/yacy/yacy_search_server.git synced 2024-09-21 00:00:13 +02:00
Code Issues Actions Packages Projects Releases Wiki Activity
4,983 Commits 1 Branch 20 Tags 836 MiB
a99a629ed4
Commit Graph

3177 Commits

Author SHA1 Message Date
low012
a99a629ed4 *) quick fix to prevent comments for blog entries which don't exist (http://forum.yacy-websuche.de/viewtopic.php?f=5&t=1554) 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5302 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-10-25 12:04:10 +00:00
low012
00e27e5050 *) fixed bug which made it possible to write files outside of the DATA/LIST directory when creating a new blacklist 
*) a blacklist will only be created if no blacklist with same name exists (some refactoring has been necessary for this)
*) further minor fixes
*) to be continued...

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5301 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-10-25 00:11:03 +00:00
danielr
0f9c0bd0d5 fix for ConcurrentModificationException at de.anomic.index.indexContainerHeap$heapCacheIterator.next(indexContainerHeap.java:324) 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5300 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-10-24 14:00:41 +00:00
danielr
103ad2a437 some javadoc 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5299 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-10-24 13:58:26 +00:00
orbiter
b098522977 some very small advances to index utf-8 (not working yet), inserted also debugging code 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5298 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-10-22 22:04:13 +00:00
orbiter
2f49666908 integrated the character decoding into the parser, removed old code 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5297 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-10-22 20:56:13 +00:00
orbiter
49293c1358 fix for deadlock in new encoder :-( 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5296 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-10-22 19:36:34 +00:00
orbiter
0edec2b760 FULL redesign of algorithms in htmlTools to encode/decode strings from/to unicode and html. 
The old process used a not really efficient way to detect html encoding strings in texts.
All calling methods had been adoped to call the new class in an enhanced way with less parameters.

Many classes in interfaces used a XML encoding only (instead of full html conversion from unicode to html); this behavior was not changed with this commit but should be controlled again since it points out possible XSS leaks

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5295 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-10-22 18:59:04 +00:00
orbiter
958ec20cd0 removed specialized umlaute-handling in html parser. This has to be replaced by something that is able to transfer all possible html encodings into utf-8. Please see SVN 5293 for test cases. 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5294 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-10-22 11:11:55 +00:00
f1ori
2e53cbc66a should compile now 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5292 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-10-22 08:50:30 +00:00
f1ori
f3bf2e379e should compile again 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5291 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-10-22 07:35:49 +00:00
f1ori
dd8441f102 fix bug: data from plasmaParser is allready converted to UTF-8 
After removing the restrictions in the code, YaCy should be able to index Unicode-charaters!


git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5290 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-10-21 20:19:10 +00:00
orbiter
6941bf42b1 performance hacks 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5288 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-10-20 14:07:09 +00:00
orbiter
9b0c4b1063 redesign of parts of the new BLOB buffer 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5287 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-10-19 22:30:44 +00:00
orbiter
1778fb420d - added some performance tweaks to the new BLOB buffer 
- removed the now superfluous HT storage thread
- reduced number of file decompression by shifting the compression moment to the future


git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5286 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-10-19 18:10:42 +00:00
orbiter
9663e61449 added another class to handle BLOB writings to the new HTCACHE data storage: 
- entries are buffered and written as stream with many entries at once (saves many IO accesses)
- entries are compressed with gzip: increases capacity of cache
- concurrency for stream-writing and compression: all writings to the cache are non-blocking

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5284 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-10-18 08:57:48 +00:00
orbiter
382226da94 fix for bug introduced in SVN 5281: parameters were switched 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5282 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-10-17 12:14:57 +00:00
danielr
f2fd043797 refactoring (moved duplicate code into methods) 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5281 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-10-17 10:39:32 +00:00
danielr
c612046e5e r5278 java 1.5 compatible 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5280 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-10-17 09:59:59 +00:00
f1ori
af71ec93bf ops, forgot to import something 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5279 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-10-16 22:44:25 +00:00
f1ori
9e65e9141c * always use UTF-8 for encoding hashes 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5278 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-10-16 22:35:27 +00:00
orbiter
826ca79735 refactoring and new architecture to store the files of the web cache: 
- files are not stored any more as individual files
- a new database structure using BLOBHeap files stores many cache entries in common files
- all file-writing procedures had been migrated to generate byte[] objects which are written with the new database methods

this is only an intermediate step to the final architecture, where cached files are written together with their metadata in one single database structure.

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5276 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-10-16 21:24:09 +00:00
danielr
f095137238 - respecting httpdMaxBusySessions (refusing new connections if limit is hit) 
- comments in serverBusyThread converted to JavaDoc
- better debug output for npe-case in diskUsage


git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5274 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-10-16 10:53:32 +00:00
orbiter
8ba33f104e fix for npe 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5269 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-10-13 21:59:53 +00:00
orbiter
998861acfd - some refactoring in BLOBHeap to enable more gap processing functions 
- better gap merging in BLOBHeap
- shrinking of heap file if gap is at end of file when file is closed

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5268 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-10-13 21:15:54 +00:00
lotus
9d50bfd0b3 fix for npe: http://forum.yacy-websuche.de/viewtopic.php?p=10562 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5267 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-10-13 09:09:53 +00:00
orbiter
766cad6e93 enhancement in memory management of BLOB Heap files / merging of deleted entries 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5266 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-10-12 22:15:01 +00:00
orbiter
7860d5d632 fix for bug in seed list management (cause was bad class overloading, only visual effects!) 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5265 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-10-12 19:51:53 +00:00
orbiter
ffed5fc415 fixed problem with lost peers in database 
migrated seedDB from BLOBTree to BLOBHeap

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5263 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-10-10 14:40:02 +00:00
orbiter
6fb865fbdc - fix of bug in iterator in kelondroBLOBHeap which caused bug in crawl profile listing 
- some refactoring of classes that use kelondroMap (Map instead of HashMap)

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5262 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-10-10 08:39:11 +00:00
orbiter
2d65887723 - fix for bug in new profile handling 
- added a new feature in ymageChart (cannot be seen yet, just wait... will be used in profiling chart)

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5261 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-10-09 22:31:43 +00:00
orbiter
ff68f394dd fix for problem with balancer and lost crawl profiles: 
if crawl profile ist lost, no robots.txt is loaded any more

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5258 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-10-08 18:26:36 +00:00
lotus
fb8d9850ea fix for http://forum.yacy-websuche.de/viewtopic.php?f=5&t=1462 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5248 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-10-05 10:03:02 +00:00
lotus
0d1a2f6183 fix for http://forum.yacy-websuche.de/viewtopic.php?f=5&t=1461 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5247 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-10-04 12:36:11 +00:00
orbiter
9ac16f565b - fixed several bugs in database management functions 
- fixed a display bug for the performance graph
- fixed deadlock when initialization of awt happens simultanously
- removed some debugging output

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5245 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-10-03 18:57:02 +00:00
orbiter
820a03f9d6 - removed some warnings 
- used fix in SVN 5233 for ysearch.java and search.java

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5237 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-10-01 20:20:39 +00:00
lotus
fe2792e9ce use accept-language header instead of user agent for language detection 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5235 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-10-01 17:47:11 +00:00
orbiter
c8bdd965ec - larger update time for status page 
- balancer writes cause of robots.txt in log file for crawl delay
- removed log output for forced GC
- smaller RAM flush for RWI cache, should cause more usage of cache and faster crawling

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5228 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-09-30 11:09:46 +00:00
lotus
dda771db9d - search result layout 
- tray only for windows

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5222 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-09-29 12:39:57 +00:00
orbiter
ce4715e305 removed indexing of anchor links and tagging such words as part of urls (that was wrong) 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5219 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-09-28 21:12:26 +00:00
orbiter
ce57de6cb3 - fixed re-setting of DHT Send/Receive settings 
- small change to network grafics: smaller circles / more URLs necessary for full radius; more PPM necessary for full crawling circles
- fixed exclusion search ('-' did not work any more)
- fixed NPE bug when FTP loader wrote to the error-db

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5218 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-09-28 20:01:10 +00:00
lotus
31c31e54e4 new tray icon image for different icon sizes (e.g. linux) 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5216 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-09-28 08:54:33 +00:00
f1ori
9589dfe080 * removed trayicon popupmenu title 
* added some menu items to trayicon


git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5213 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-09-27 08:25:16 +00:00
lotus
5a637f004d localized tray 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5212 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-09-26 11:09:54 +00:00
lotus
9d4f0325e1 - removed shutdown from search page (we have it in tray now!) 
- fixed doubleclick action for tray

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5211 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-09-26 10:55:08 +00:00
lotus
214277dad6 - revert r5202 
- cleanup
- installer checks for JRE 1.6 only

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5210 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-09-26 06:52:36 +00:00
f1ori
7afa084207 * add nativ java trayicon, using reflections 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5209 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-09-25 19:36:49 +00:00
apfelmaennchen
b97ff24b43 bookmarksDB / xbel.xml: 
- added support for folder=/foldername
- it crashes if foldername ends with /

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5207 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-09-24 21:16:13 +00:00
orbiter
6e7d113eac fix for wrong index initialization after network switch 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5203 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-09-23 23:30:25 +00:00
lotus
0a0cc3bf67 added missing classes to build target "run" 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5201 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-09-23 15:54:12 +00:00