Marown/yacy_search_server
1
0
Fork 0
mirror of https://github.com/yacy/yacy_search_server.git synced 2024-09-21 00:00:13 +02:00
Code Issues Actions Packages Projects Releases Wiki Activity
2,639 Commits 1 Branch 20 Tags 836 MiB
e9afe39cbb
Commit Graph

1748 Commits

Author SHA1 Message Date
orbiter
c40fca08a2 fixed bad handling of string separation 
you can now use a new encoding attribute to create strings from byte arrays

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2693 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2006-10-02 10:21:14 +00:00
orbiter
5a40ea7866 refactoring of wget string list generation 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2692 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2006-10-02 09:59:20 +00:00
orbiter
dbc2e039bb added time-out option parameter to call hierarchy 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2691 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2006-10-02 09:40:18 +00:00
orbiter
d4c239e4be - fixed problem in collection index with deletion of single url references 
- added automatic deletion of not-found snippets after search

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2689 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2006-10-02 01:40:52 +00:00
orbiter
00746ca232 identified and fixed search performance problem caused by 
snippet loading. Some access to header-db had been twice and even
more times in some cases. Snippet resource loading fixed.
Furthermore the snippet loading during remote search within the
remote peer has been disabled, but can be switched on remotely by
new flag 'includesnippet=true'

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2688 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2006-10-02 01:15:02 +00:00
orbiter
b033a80750 better control of failure in node seek of kelondroTree 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2686 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2006-10-02 00:13:19 +00:00
orbiter
310f1c41cd added option to see ranking scores in surftipps 
and some cleanups

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2684 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2006-09-30 23:28:03 +00:00
theli
a2e3095044 *) Bugfix. Add missing plasmaParserDocument.close() calls 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2680 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2006-09-30 10:09:01 +00:00
theli
cd5f349666 *) Better handling of large files during parsing 
Extracted text of files that are larger than 5MB is stored in a temp file instead of keeping it in memory
*) plasmaParserDocument.java; getText now returnes an inputStream instead of a byte array
*) plasmaParserDocument.java: new function getTextBytes returns the parsed content as byte array
   Attention: the caller of this function has to ensure that enough memory is available to do this 
   to avoid OutOfMemory Exceptions
*) httpd.java: better error handling if the soaphander is not installed
*) pdfParser.java: 
   - better handling of documents with exotic charsets
   - better handling of large documents
   - better error logging of encrypted documents
*) rtfParser.java: Bugfix for UTF-8 support
*) tarParser.java: better handling of large documents
*) zipParser.java: better handling of large documents
*) plasmaCrawlEURL.java: new errorcode for encrypted documents
*) plasmaParserDocument.java: the extracted text can now be passed
   to this object as byte array or temp file   

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2679 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2006-09-30 09:31:53 +00:00
theli
8b2ceddb91 *) Displaying servere and warning logging messages in different colors on ViewLog_p.html 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2678 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2006-09-30 08:12:22 +00:00
low012
f8ac694e51 *) fixed a bug where searchword in snippets were not displayed bold in front of a punctuation mark (see http://www.yacy-forum.de/viewtopic.php?p=25998) 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2677 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2006-09-30 00:27:42 +00:00
orbiter
df1629b05a - code cleanup 
- version 0.471
- moved surftipps to own web page


git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2676 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2006-09-29 22:27:20 +00:00
theli
c665f6cddb *) handling of quotes in charset string 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2674 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2006-09-28 06:29:15 +00:00
theli
b73efd5565 *) missing changes needed because of last commit 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2673 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2006-09-28 05:48:28 +00:00
theli
140ddba93f *) adding soap functions to pause and resume the crawler 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2668 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2006-09-27 05:22:43 +00:00
orbiter
2463e5624a 'quick' release 0.47 
- documentation update
- necessary bugfixes (missing css for new peers)
- reduced effect of search result redundancy filter
- removed some debug output, but not all

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2665 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2006-09-26 23:41:54 +00:00
theli
49fbb688df *) SOAP: old urlInfo renamed to urlInfoByHash, new urlInfo Function added. 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2662 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2006-09-26 15:14:33 +00:00
theli
8f143d516b *) make snippet fetcher accessible via soap api 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2661 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2006-09-26 15:07:16 +00:00
theli
97615af406 *) Restructuring of YaCy SOAP services 
- general functions moved to abstract service class
   - service class splitted into SearchService, CrawlService, StatusService
*) Bugfix for SOAP search services
   - Attention: some xml tages where renamed
   See: http://www.yacy-forum.de/viewtopic.php?p=25877
*) New SOAP service function urlInfo to view the parsed content of an URL
   See: http://www.yacy-forum.de/viewtopic.php?p=25869

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2660 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2006-09-26 14:47:44 +00:00
theli
241b881560 *) Redesign of YaCy SOAP handler 
- should be more fail-safe now
   - better handling of compressed request bodies
   - better handling of persistent connections
   - better handling of AxisFaults

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2659 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2006-09-26 12:24:40 +00:00
theli
009a33170b *) Content-Location header added 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2658 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2006-09-26 04:32:01 +00:00
theli
1aa07a52cd *) Bugfix for UnsupportedEncodingException if the media type contains multiple parameters 
See: http://www.yacy-forum.de/viewtopic.php?p=25832#25826

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2654 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2006-09-24 15:50:51 +00:00
theli
625c2ce6b1 *) bugfix for snippet fetching problem if content but not http header is available in cache 
See: http://www.yacy-forum.de/viewtopic.php?p=25748

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2651 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2006-09-22 11:55:28 +00:00
theli
813a8a8179 *) migration of mimeTypeParser to jmimemagic 0.1 
- better mimetype detection for rss feeds
   - better mimetype detection for odt documents (less memory consuming)
   - two new detector classes implementing MagicDetector interface of jmimemagic

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2650 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2006-09-22 11:40:46 +00:00
hermens
3f5a4153a0 Make Peers more receptible to transferred indexes 
- Set MaxWordCount for dhtInCache to indexDistribution.dhtReceiptLimit
  so that the inCache gets flushed when the limit is passed
- Modify flushCacheSome to flush enough words to get below MaxWordCount immediately



git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2649 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2006-09-22 10:58:58 +00:00
theli
57415b6889 *) Bugfix for surftipps UTF-8 problem 
See: http://www.yacy-forum.de/viewtopic.php?t=2864

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2647 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2006-09-22 05:40:29 +00:00
allo
b0a4fcce8c fix from theli 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2642 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2006-09-20 18:03:24 +00:00
theli
b6c7b91582 *) Parser now throws an ParserException instead of returning null on parsing errors (e.g. needed by snippet fetcher) 
*) better logging of parser failures
*) simplified usage of plasmaparser through switchboard
*) restructuring of crawler
   - crawler now returns an error message if it is used in sync mode (e.g. by snippet fetcher)
*) snippet-fetcher: more verbose error messages
*) serverByteBuffer.java: adding new function append(String,encoding)
*) serverFileUtils.java: adding functions to copy only a given number of bytes between streams


git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2641 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2006-09-20 12:25:07 +00:00
orbiter
e03427871e enhanced surftipps: 
- added switchh to show or hide surftipps
- more news contribute to surftipps
- added voting system for surftipps

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2638 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2006-09-20 07:17:41 +00:00
theli
1dc12d6659 *) Bugfix for shutdown problem caused by cacheScan thread 
See: http://www.yacy-forum.de/viewtopic.php?p=25729

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2636 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2006-09-20 04:36:25 +00:00
borg-0300
42173462f5 rename cutUrlText to shortenURLString; 
other little things;

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2635 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2006-09-19 20:47:45 +00:00
borg-0300
af1d89e381 check url == null added; 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2634 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2006-09-19 20:12:26 +00:00
theli
cc667b0aa5 *) htmlFilterContentScraper.java: adding support for link tag 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2633 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2006-09-19 16:13:13 +00:00
theli
26dfbb7499 *) Bugfix for UTF-8: url names are now stored properly in stackcrawl, crawler, indexing queue and should be displayed correct on the gui 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2630 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2006-09-19 05:19:41 +00:00
theli
cf6acff2c2 *) Bugfix. htmlFilterInputStream document analysis did not work properly for documents smaller than the 
default InputStream Buffer size.

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2629 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2006-09-19 04:58:34 +00:00
borg-0300
f18304ddd3 unused/not needed imports removes; 
properties added;

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2628 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2006-09-18 22:21:18 +00:00
orbiter
ec031eb993 first version of surftipps 
see http://localhost:8080/index.html

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2627 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2006-09-18 20:14:21 +00:00
borg-0300
b174fbd0ca "import ...*" removed; 
properties added;

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2626 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2006-09-18 18:31:27 +00:00
orbiter
807756150e patch for strange bug reported by email 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2625 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2006-09-18 16:50:31 +00:00
theli
5c6251bced *) some improvements for extended html document charset support 
- new class htmlFilterInputStream.java which allows to pre-analyze the html header to extract
     the charset meta data. This is only enabled for the crawler at the moment. Integration into 
     proxy needs more testing.     
   - adding eventlisterner interfaces to the htmlscraper to allow other classes to get informed
     about detected tags (used by the htmlFilterInputStream.java)

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2624 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2006-09-18 15:36:04 +00:00
theli
33f0f703c0 *) reinserting type cast again 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2623 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2006-09-18 13:21:12 +00:00
orbiter
8c11a543dc fixed line ending coding 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2622 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2006-09-18 13:17:31 +00:00
theli
b690597275 *) adding casts to avoid compatibility problems between java 1.4 and java 1.5 writer class usage 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2621 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2006-09-18 12:17:51 +00:00
theli
5afb0cbce8 *) setting default charset (for unkown documents) to iso-8859-1 
*)

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2620 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2006-09-18 11:39:06 +00:00
orbiter
f453c14b5d removed unreacheable catch blocks and unused imports 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2619 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2006-09-18 11:23:58 +00:00
theli
ad7f600f25 *) Bugfix. re-enabling inheritance of serverCharBuffer from writer class 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2618 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2006-09-18 11:04:16 +00:00
theli
97d2a08ef1 *) restructuring needed to support parsing of documents using various charsets 
- serverFileUtils.java: 
   -- adding methods to copy from stream to writer and readers to writers
   -- moving httpc writeX methods into serverFileUtils class
   - serverCharBuffer.java: removing inheritance from Writer class
   - replacing htmlFilterOutputStream by htmlFilterWriter class which handles
     content as char stream
   - htmlFilterContentTransformer.java: deactivating getText mode 
    (still needs to be migrated to use char streams instead of byte streams)
   - changes in several classes to use htmlFilterWriter instead of htmlFilterOutputStream
   - changes in Scraper and Transformer classes to operate on chars instead of bytes
   - httpdProxyHandler.java: bugfix. clientTimeout setting was missing in config file

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2617 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2006-09-18 10:12:11 +00:00
theli
fc594e8eda *) adding httpContentLengthInputStream.java class to allow reading of http response bodies 
until EOF even if a persistent connection is used
*) httpdByteCountInputStream.java: adding skip method
*) httpHeader.java: adding getCharacterEncoding function

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2616 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2006-09-18 10:00:28 +00:00
low012
cd636eb00e *) Fix for the fix... 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2615 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2006-09-18 01:24:26 +00:00
low012
f9a5b55a9e *) Fixed bug described in http://www.yacy-forum.de/viewtopic.php?p=25448#25448 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2614 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2006-09-18 01:19:54 +00:00
orbiter
3aac5b26da - added automatic tag generation when a web page from the search results is added 
- added new image 'B' in front of search results for bookmark generation
- added news generation when a public bookmark is added
- the '+' in front of search results has new meaning: positive rating for that result
- added news generation when a '+' is hit

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2613 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2006-09-18 00:37:02 +00:00
low012
8a30c5343d *) Fixed bug where exclamation marks could get lost between [=...=] and <pre>...</pre> 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2612 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2006-09-17 23:42:36 +00:00
low012
d8f4b17e31 *) Hopefully fixed bug described in http://www.yacy-forum.de/viewtopic.php?t=2825. 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2611 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2006-09-17 22:57:10 +00:00
theli
0e84a969d6 *) Bugfix for serverCharBuffer read from file operation 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2607 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2006-09-16 13:11:32 +00:00
theli
90ef19d778 *) first version of a serverCharBuffer 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2606 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2006-09-16 12:56:03 +00:00
orbiter
d374ef2bbe bugfix for tryRemoveURLs 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2605 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2006-09-16 00:34:34 +00:00
orbiter
f644a1c3a7 better evaluation of index abstracts 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2604 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2006-09-16 00:07:09 +00:00
orbiter
1b48473bc5 bugfix to utf8 recognition 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2603 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2006-09-15 23:55:06 +00:00
orbiter
90f7241b59 serverByteBuffer.trim() can now recognize utf-8 characters 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2602 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2006-09-15 23:52:26 +00:00
allo
2fd610b556 http://www.yacy-forum.de/viewtopic.php?p=25611#25611 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2601 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2006-09-15 17:48:41 +00:00
theli
e34d9b3fec *) charset aware headlines (after the serverByteBuffer.trim problem is solved) 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2599 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2006-09-15 15:07:35 +00:00
theli
8115ac47b5 *) charset aware metadata parsing 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2598 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2006-09-15 15:01:25 +00:00
theli
3ac30bdf22 *) some todo markers added for additional charset support 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2597 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2006-09-15 14:49:43 +00:00
theli
06fa891152 *) htmlFilterContentScraper.java: using proper charset for document title 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2595 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2006-09-15 14:05:28 +00:00
theli
74c3e7cf29 *) storing document charset into plasmaParserDocument object (is needed later by the condenser) 
*) htmlFilterContentScraper.java: using proper charset for document title
*) serverByteBuffer.java: adding new toString which allows to specify the charset for byte encoding


git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2593 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2006-09-15 13:18:12 +00:00
theli
c5d3020941 *) better errorhandling for last commit 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2592 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2006-09-15 12:56:01 +00:00
theli
d0a5a53789 *) changes needed for multi-language support 
- parsers may need to know the charset of the byte stream 

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2591 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2006-09-15 12:52:46 +00:00
orbiter
d82875c72b removed removal of 'funny symbols' that may have caused utf-8 problems 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2589 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2006-09-15 09:08:15 +00:00
orbiter
26ab1fa885 fixed null pointer exception 
See http://www.yacy-forum.de/viewtopic.php?p=25598#25598

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2588 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2006-09-15 08:50:16 +00:00
theli
b0e8ff6eda *) some TODO makers for UTF-8 problem 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2586 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2006-09-15 05:31:30 +00:00
orbiter
41e27b85b7 fix for crawler condition 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2583 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2006-09-15 00:38:45 +00:00
orbiter
0ee7e45413 bugfix for merge method (caused by bad refactoring) 
see http://www.yacy-forum.de/viewtopic.php?p=25529#25529

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2581 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2006-09-14 10:30:25 +00:00
orbiter
5c2f30eaca adjustments to dhtInCache write 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2579 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2006-09-14 09:28:17 +00:00
theli
9ecf7f0da2 *) some TODO makers for UTF-8 problem 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2578 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2006-09-14 05:37:46 +00:00
theli
e2f8339827 *) some bugfixes for UTF-8 related problems 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2577 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2006-09-14 05:16:36 +00:00
orbiter
c89d8142bb replaced old 'kCache' by a full-controlled cache 
there are now two full-controlled caches for incoming indexes:
- dhtIn
- dhtOut
during indexing, all indexes that shall not be transported to remote peers
because they belong to the own peer are stored to dhtIn. It is furthermore
ensured that received indexes are not again transmitted to other peers
directly. They may, however be transmitted later if the network grows.

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2574 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2006-09-14 00:51:02 +00:00
orbiter
6e2907135a bugfixes for remote search server part 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2573 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2006-09-13 22:19:34 +00:00
orbiter
cf9884e22b first attempt to implement a secondary search 
this is a set of search processes that shall enrich search results
with specialized requests to realize a combination of search results
from different peers.

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2571 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2006-09-13 17:13:28 +00:00
theli
2a06ce5538 *) next bugfix for UTF-8 
- Sending UFT-8 messages to other peers did not work
   - httpd.java: minor corrections for UTF-8

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2570 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2006-09-13 15:47:56 +00:00
theli
bdc51591ae *) UTF-8 Bug solved (hopefully) 
See: http://www.yacy-forum.de/viewtopic.php?p=25522

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2569 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2006-09-13 14:48:58 +00:00
theli
ef751b9d33 *) removing all string operations from the template engine 
- engine should fully operate on bytes now

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2567 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2006-09-13 13:56:10 +00:00
orbiter
7ef80c1026 more debugging 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2566 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2006-09-13 13:52:46 +00:00
orbiter
b251076e64 avoid ConcurrentModificationException 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2563 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2006-09-13 10:36:18 +00:00
orbiter
75b198bc02 - updated references to indexContainer 
- more bugfixes and debugging for indexAbstract processing

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2555 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2006-09-12 11:13:27 +00:00
orbiter
0bed3b9ac3 removed superfluous interface 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2554 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2006-09-12 11:09:51 +00:00
orbiter
b7e7808ea6 wordmigration now works also for new index database 
if the new database is switched on, no 'too big' messages appear,
all the WORDS files can be completely migrated

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2553 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2006-09-12 08:23:47 +00:00
theli
a0ddf2ec11 *) AbstractCrawlWorker.java: delete already downloaded data on crawling error 
*) plasmaSwitchboard.java: log unexpected errors while parsing/indexing

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2552 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2006-09-12 04:50:12 +00:00
orbiter
4f9e42d5ed more changes towards better join-search 
- fixed problems with index-abstract generation
- added analysis output for index abstract receive

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2551 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2006-09-12 00:42:42 +00:00
orbiter
a7281a9b4d fix for last commit 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2545 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2006-09-11 11:12:42 +00:00
orbiter
82a6054275 - fixed bug with new indexAbstract generation 
- added partly evaluation of indexAbstracts during remote searches

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2544 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2006-09-11 10:39:25 +00:00
theli
fded1f4a5d *) better handling of maximum file size limit in crawler 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2543 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2006-09-11 08:26:39 +00:00
orbiter
416b4e5c6b ups 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2542 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2006-09-11 08:17:55 +00:00
orbiter
309accb983 memory control for ymage generation: 
the ymageMatrix initializer throws an RuntimeException if there is not
enough memory available to generate a new ymage of wanted size

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2541 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2006-09-11 07:01:39 +00:00
orbiter
74d1dea30b changes towards better join-search 
- added generation of a compressed index within remote peers during global search
- added selection of specific urls within remote peers during secondary global search


git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2539 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2006-09-10 22:36:47 +00:00
orbiter
ae4e8ce03e - cut for 'probably last html-interface version': version number update 
- small enhancement to ranking

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2536 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2006-09-10 19:44:11 +00:00
orbiter
64bed59ee8 enhancements to ranking 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2535 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2006-09-09 23:44:54 +00:00
theli
63893003be *) Adding settings page for the crawler which allows to specify a file size limit and the timeout to use. 
*) adding first version of maximum filesize check for the crawler

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2534 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2006-09-09 15:06:49 +00:00
auron_x
06b1365066 *) fixed existing protection against divbyzero and removed the new one 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2530 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2006-09-08 23:43:30 +00:00
orbiter
94d7ced900 fix for last ranking commit 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2529 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2006-09-08 21:14:57 +00:00
orbiter
cc97a3e9c6 fixed possibly bug with indexOutOfBoundsException 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2528 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2006-09-08 20:27:45 +00:00