Marown/yacy_search_server
1
0
Fork 0
mirror of https://github.com/yacy/yacy_search_server.git synced 2024-09-21 00:00:13 +02:00
Code Issues Actions Packages Projects Releases Wiki Activity
2,540 Commits 1 Branch 20 Tags 836 MiB
b0cab1e731
Commit Graph

1762 Commits

Author SHA1 Message Date
theli
1aa07a52cd *) Bugfix for UnsupportedEncodingException if the media type contains multiple parameters 
See: http://www.yacy-forum.de/viewtopic.php?p=25832#25826

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2654 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2006-09-24 15:50:51 +00:00
theli
625c2ce6b1 *) bugfix for snippet fetching problem if content but not http header is available in cache 
See: http://www.yacy-forum.de/viewtopic.php?p=25748

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2651 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2006-09-22 11:55:28 +00:00
theli
813a8a8179 *) migration of mimeTypeParser to jmimemagic 0.1 
- better mimetype detection for rss feeds
   - better mimetype detection for odt documents (less memory consuming)
   - two new detector classes implementing MagicDetector interface of jmimemagic

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2650 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2006-09-22 11:40:46 +00:00
hermens
3f5a4153a0 Make Peers more receptible to transferred indexes 
- Set MaxWordCount for dhtInCache to indexDistribution.dhtReceiptLimit
  so that the inCache gets flushed when the limit is passed
- Modify flushCacheSome to flush enough words to get below MaxWordCount immediately



git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2649 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2006-09-22 10:58:58 +00:00
theli
57415b6889 *) Bugfix for surftipps UTF-8 problem 
See: http://www.yacy-forum.de/viewtopic.php?t=2864

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2647 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2006-09-22 05:40:29 +00:00
allo
b0a4fcce8c fix from theli 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2642 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2006-09-20 18:03:24 +00:00
theli
b6c7b91582 *) Parser now throws an ParserException instead of returning null on parsing errors (e.g. needed by snippet fetcher) 
*) better logging of parser failures
*) simplified usage of plasmaparser through switchboard
*) restructuring of crawler
   - crawler now returns an error message if it is used in sync mode (e.g. by snippet fetcher)
*) snippet-fetcher: more verbose error messages
*) serverByteBuffer.java: adding new function append(String,encoding)
*) serverFileUtils.java: adding functions to copy only a given number of bytes between streams


git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2641 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2006-09-20 12:25:07 +00:00
theli
64b2ef5aae *) Trying to bugfix shutdown problem 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2639 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2006-09-20 10:13:23 +00:00
orbiter
e03427871e enhanced surftipps: 
- added switchh to show or hide surftipps
- more news contribute to surftipps
- added voting system for surftipps

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2638 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2006-09-20 07:17:41 +00:00
theli
1dc12d6659 *) Bugfix for shutdown problem caused by cacheScan thread 
See: http://www.yacy-forum.de/viewtopic.php?p=25729

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2636 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2006-09-20 04:36:25 +00:00
borg-0300
42173462f5 rename cutUrlText to shortenURLString; 
other little things;

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2635 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2006-09-19 20:47:45 +00:00
borg-0300
af1d89e381 check url == null added; 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2634 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2006-09-19 20:12:26 +00:00
theli
cc667b0aa5 *) htmlFilterContentScraper.java: adding support for link tag 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2633 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2006-09-19 16:13:13 +00:00
theli
26dfbb7499 *) Bugfix for UTF-8: url names are now stored properly in stackcrawl, crawler, indexing queue and should be displayed correct on the gui 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2630 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2006-09-19 05:19:41 +00:00
theli
cf6acff2c2 *) Bugfix. htmlFilterInputStream document analysis did not work properly for documents smaller than the 
default InputStream Buffer size.

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2629 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2006-09-19 04:58:34 +00:00
borg-0300
f18304ddd3 unused/not needed imports removes; 
properties added;

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2628 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2006-09-18 22:21:18 +00:00
orbiter
ec031eb993 first version of surftipps 
see http://localhost:8080/index.html

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2627 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2006-09-18 20:14:21 +00:00
borg-0300
b174fbd0ca "import ...*" removed; 
properties added;

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2626 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2006-09-18 18:31:27 +00:00
orbiter
807756150e patch for strange bug reported by email 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2625 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2006-09-18 16:50:31 +00:00
theli
5c6251bced *) some improvements for extended html document charset support 
- new class htmlFilterInputStream.java which allows to pre-analyze the html header to extract
     the charset meta data. This is only enabled for the crawler at the moment. Integration into 
     proxy needs more testing.     
   - adding eventlisterner interfaces to the htmlscraper to allow other classes to get informed
     about detected tags (used by the htmlFilterInputStream.java)

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2624 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2006-09-18 15:36:04 +00:00
theli
33f0f703c0 *) reinserting type cast again 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2623 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2006-09-18 13:21:12 +00:00
orbiter
8c11a543dc fixed line ending coding 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2622 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2006-09-18 13:17:31 +00:00
theli
b690597275 *) adding casts to avoid compatibility problems between java 1.4 and java 1.5 writer class usage 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2621 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2006-09-18 12:17:51 +00:00
theli
5afb0cbce8 *) setting default charset (for unkown documents) to iso-8859-1 
*)

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2620 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2006-09-18 11:39:06 +00:00
orbiter
f453c14b5d removed unreacheable catch blocks and unused imports 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2619 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2006-09-18 11:23:58 +00:00
theli
ad7f600f25 *) Bugfix. re-enabling inheritance of serverCharBuffer from writer class 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2618 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2006-09-18 11:04:16 +00:00
theli
97d2a08ef1 *) restructuring needed to support parsing of documents using various charsets 
- serverFileUtils.java: 
   -- adding methods to copy from stream to writer and readers to writers
   -- moving httpc writeX methods into serverFileUtils class
   - serverCharBuffer.java: removing inheritance from Writer class
   - replacing htmlFilterOutputStream by htmlFilterWriter class which handles
     content as char stream
   - htmlFilterContentTransformer.java: deactivating getText mode 
    (still needs to be migrated to use char streams instead of byte streams)
   - changes in several classes to use htmlFilterWriter instead of htmlFilterOutputStream
   - changes in Scraper and Transformer classes to operate on chars instead of bytes
   - httpdProxyHandler.java: bugfix. clientTimeout setting was missing in config file

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2617 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2006-09-18 10:12:11 +00:00
theli
fc594e8eda *) adding httpContentLengthInputStream.java class to allow reading of http response bodies 
until EOF even if a persistent connection is used
*) httpdByteCountInputStream.java: adding skip method
*) httpHeader.java: adding getCharacterEncoding function

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2616 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2006-09-18 10:00:28 +00:00
low012
cd636eb00e *) Fix for the fix... 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2615 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2006-09-18 01:24:26 +00:00
low012
f9a5b55a9e *) Fixed bug described in http://www.yacy-forum.de/viewtopic.php?p=25448#25448 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2614 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2006-09-18 01:19:54 +00:00
orbiter
3aac5b26da - added automatic tag generation when a web page from the search results is added 
- added new image 'B' in front of search results for bookmark generation
- added news generation when a public bookmark is added
- the '+' in front of search results has new meaning: positive rating for that result
- added news generation when a '+' is hit

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2613 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2006-09-18 00:37:02 +00:00
low012
8a30c5343d *) Fixed bug where exclamation marks could get lost between [=...=] and <pre>...</pre> 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2612 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2006-09-17 23:42:36 +00:00
low012
d8f4b17e31 *) Hopefully fixed bug described in http://www.yacy-forum.de/viewtopic.php?t=2825. 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2611 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2006-09-17 22:57:10 +00:00
theli
0e84a969d6 *) Bugfix for serverCharBuffer read from file operation 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2607 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2006-09-16 13:11:32 +00:00
theli
90ef19d778 *) first version of a serverCharBuffer 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2606 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2006-09-16 12:56:03 +00:00
orbiter
d374ef2bbe bugfix for tryRemoveURLs 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2605 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2006-09-16 00:34:34 +00:00
orbiter
f644a1c3a7 better evaluation of index abstracts 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2604 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2006-09-16 00:07:09 +00:00
orbiter
1b48473bc5 bugfix to utf8 recognition 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2603 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2006-09-15 23:55:06 +00:00
orbiter
90f7241b59 serverByteBuffer.trim() can now recognize utf-8 characters 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2602 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2006-09-15 23:52:26 +00:00
allo
2fd610b556 http://www.yacy-forum.de/viewtopic.php?p=25611#25611 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2601 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2006-09-15 17:48:41 +00:00
theli
e34d9b3fec *) charset aware headlines (after the serverByteBuffer.trim problem is solved) 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2599 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2006-09-15 15:07:35 +00:00
theli
8115ac47b5 *) charset aware metadata parsing 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2598 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2006-09-15 15:01:25 +00:00
theli
3ac30bdf22 *) some todo markers added for additional charset support 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2597 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2006-09-15 14:49:43 +00:00
theli
06fa891152 *) htmlFilterContentScraper.java: using proper charset for document title 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2595 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2006-09-15 14:05:28 +00:00
theli
74c3e7cf29 *) storing document charset into plasmaParserDocument object (is needed later by the condenser) 
*) htmlFilterContentScraper.java: using proper charset for document title
*) serverByteBuffer.java: adding new toString which allows to specify the charset for byte encoding


git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2593 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2006-09-15 13:18:12 +00:00
theli
c5d3020941 *) better errorhandling for last commit 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2592 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2006-09-15 12:56:01 +00:00
theli
d0a5a53789 *) changes needed for multi-language support 
- parsers may need to know the charset of the byte stream 

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2591 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2006-09-15 12:52:46 +00:00
orbiter
d82875c72b removed removal of 'funny symbols' that may have caused utf-8 problems 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2589 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2006-09-15 09:08:15 +00:00
orbiter
26ab1fa885 fixed null pointer exception 
See http://www.yacy-forum.de/viewtopic.php?p=25598#25598

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2588 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2006-09-15 08:50:16 +00:00
theli
b0e8ff6eda *) some TODO makers for UTF-8 problem 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2586 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2006-09-15 05:31:30 +00:00