1738 Commits

Author	SHA1	Message	Date
theli	e34d9b3fec	*) charset aware headlines (after the serverByteBuffer.trim problem is solved) ... git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2599 6c8d7289-2bf4-0310-a012-ef5d649a1542	2006-09-15 15:07:35 +00:00
theli	8115ac47b5	*) charset aware metadata parsing ... git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2598 6c8d7289-2bf4-0310-a012-ef5d649a1542	2006-09-15 15:01:25 +00:00
theli	3ac30bdf22	*) some todo markers added for additional charset support ... git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2597 6c8d7289-2bf4-0310-a012-ef5d649a1542	2006-09-15 14:49:43 +00:00
theli	06fa891152	*) htmlFilterContentScraper.java: using proper charset for document title ... git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2595 6c8d7289-2bf4-0310-a012-ef5d649a1542	2006-09-15 14:05:28 +00:00
theli	74c3e7cf29	*) storing document charset into plasmaParserDocument object (is needed later by the condenser) ... *) htmlFilterContentScraper.java: using proper charset for document title *) serverByteBuffer.java: adding new toString which allows to specify the charset for byte encoding git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2593 6c8d7289-2bf4-0310-a012-ef5d649a1542	2006-09-15 13:18:12 +00:00
theli	c5d3020941	*) better errorhandling for last commit ... git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2592 6c8d7289-2bf4-0310-a012-ef5d649a1542	2006-09-15 12:56:01 +00:00
theli	d0a5a53789	*) changes needed for multi-language support ... - parsers may need to know the charset of the byte stream git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2591 6c8d7289-2bf4-0310-a012-ef5d649a1542	2006-09-15 12:52:46 +00:00
orbiter	d82875c72b	removed removal of 'funny symbols' that may have caused utf-8 problems ... git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2589 6c8d7289-2bf4-0310-a012-ef5d649a1542	2006-09-15 09:08:15 +00:00
orbiter	26ab1fa885	fixed null pointer exception ... See http://www.yacy-forum.de/viewtopic.php?p=25598#25598 git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2588 6c8d7289-2bf4-0310-a012-ef5d649a1542	2006-09-15 08:50:16 +00:00
theli	b0e8ff6eda	*) some TODO makers for UTF-8 problem ... git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2586 6c8d7289-2bf4-0310-a012-ef5d649a1542	2006-09-15 05:31:30 +00:00
orbiter	41e27b85b7	fix for crawler condition ... git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2583 6c8d7289-2bf4-0310-a012-ef5d649a1542	2006-09-15 00:38:45 +00:00
orbiter	0ee7e45413	bugfix for merge method (caused by bad refactoring) ... see http://www.yacy-forum.de/viewtopic.php?p=25529#25529 git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2581 6c8d7289-2bf4-0310-a012-ef5d649a1542	2006-09-14 10:30:25 +00:00
orbiter	5c2f30eaca	adjustments to dhtInCache write ... git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2579 6c8d7289-2bf4-0310-a012-ef5d649a1542	2006-09-14 09:28:17 +00:00
theli	9ecf7f0da2	*) some TODO makers for UTF-8 problem ... git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2578 6c8d7289-2bf4-0310-a012-ef5d649a1542	2006-09-14 05:37:46 +00:00
theli	e2f8339827	*) some bugfixes for UTF-8 related problems ... git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2577 6c8d7289-2bf4-0310-a012-ef5d649a1542	2006-09-14 05:16:36 +00:00
orbiter	c89d8142bb	replaced old 'kCache' by a full-controlled cache ... there are now two full-controlled caches for incoming indexes: - dhtIn - dhtOut during indexing, all indexes that shall not be transported to remote peers because they belong to the own peer are stored to dhtIn. It is furthermore ensured that received indexes are not again transmitted to other peers directly. They may, however be transmitted later if the network grows. git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2574 6c8d7289-2bf4-0310-a012-ef5d649a1542	2006-09-14 00:51:02 +00:00
orbiter	6e2907135a	bugfixes for remote search server part ... git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2573 6c8d7289-2bf4-0310-a012-ef5d649a1542	2006-09-13 22:19:34 +00:00
orbiter	cf9884e22b	first attempt to implement a secondary search ... this is a set of search processes that shall enrich search results with specialized requests to realize a combination of search results from different peers. git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2571 6c8d7289-2bf4-0310-a012-ef5d649a1542	2006-09-13 17:13:28 +00:00
theli	2a06ce5538	*) next bugfix for UTF-8 ... - Sending UFT-8 messages to other peers did not work - httpd.java: minor corrections for UTF-8 git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2570 6c8d7289-2bf4-0310-a012-ef5d649a1542	2006-09-13 15:47:56 +00:00
theli	bdc51591ae	*) UTF-8 Bug solved (hopefully) ... See: http://www.yacy-forum.de/viewtopic.php?p=25522 git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2569 6c8d7289-2bf4-0310-a012-ef5d649a1542	2006-09-13 14:48:58 +00:00
theli	ef751b9d33	*) removing all string operations from the template engine ... - engine should fully operate on bytes now git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2567 6c8d7289-2bf4-0310-a012-ef5d649a1542	2006-09-13 13:56:10 +00:00
orbiter	7ef80c1026	more debugging ... git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2566 6c8d7289-2bf4-0310-a012-ef5d649a1542	2006-09-13 13:52:46 +00:00
orbiter	b251076e64	avoid ConcurrentModificationException ... git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2563 6c8d7289-2bf4-0310-a012-ef5d649a1542	2006-09-13 10:36:18 +00:00
orbiter	75b198bc02	- updated references to indexContainer ... - more bugfixes and debugging for indexAbstract processing git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2555 6c8d7289-2bf4-0310-a012-ef5d649a1542	2006-09-12 11:13:27 +00:00
orbiter	0bed3b9ac3	removed superfluous interface ... git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2554 6c8d7289-2bf4-0310-a012-ef5d649a1542	2006-09-12 11:09:51 +00:00
orbiter	b7e7808ea6	wordmigration now works also for new index database ... if the new database is switched on, no 'too big' messages appear, all the WORDS files can be completely migrated git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2553 6c8d7289-2bf4-0310-a012-ef5d649a1542	2006-09-12 08:23:47 +00:00
theli	a0ddf2ec11	*) AbstractCrawlWorker.java: delete already downloaded data on crawling error ... *) plasmaSwitchboard.java: log unexpected errors while parsing/indexing git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2552 6c8d7289-2bf4-0310-a012-ef5d649a1542	2006-09-12 04:50:12 +00:00
orbiter	4f9e42d5ed	more changes towards better join-search ... - fixed problems with index-abstract generation - added analysis output for index abstract receive git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2551 6c8d7289-2bf4-0310-a012-ef5d649a1542	2006-09-12 00:42:42 +00:00
orbiter	a7281a9b4d	fix for last commit ... git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2545 6c8d7289-2bf4-0310-a012-ef5d649a1542	2006-09-11 11:12:42 +00:00
orbiter	82a6054275	- fixed bug with new indexAbstract generation ... - added partly evaluation of indexAbstracts during remote searches git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2544 6c8d7289-2bf4-0310-a012-ef5d649a1542	2006-09-11 10:39:25 +00:00
theli	fded1f4a5d	*) better handling of maximum file size limit in crawler ... git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2543 6c8d7289-2bf4-0310-a012-ef5d649a1542	2006-09-11 08:26:39 +00:00
orbiter	416b4e5c6b	ups ... git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2542 6c8d7289-2bf4-0310-a012-ef5d649a1542	2006-09-11 08:17:55 +00:00
orbiter	309accb983	memory control for ymage generation: ... the ymageMatrix initializer throws an RuntimeException if there is not enough memory available to generate a new ymage of wanted size git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2541 6c8d7289-2bf4-0310-a012-ef5d649a1542	2006-09-11 07:01:39 +00:00
orbiter	74d1dea30b	changes towards better join-search ... - added generation of a compressed index within remote peers during global search - added selection of specific urls within remote peers during secondary global search git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2539 6c8d7289-2bf4-0310-a012-ef5d649a1542	2006-09-10 22:36:47 +00:00
orbiter	ae4e8ce03e	- cut for 'probably last html-interface version': version number update ... - small enhancement to ranking git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2536 6c8d7289-2bf4-0310-a012-ef5d649a1542	2006-09-10 19:44:11 +00:00
orbiter	64bed59ee8	enhancements to ranking ... git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2535 6c8d7289-2bf4-0310-a012-ef5d649a1542	2006-09-09 23:44:54 +00:00
theli	63893003be	*) Adding settings page for the crawler which allows to specify a file size limit and the timeout to use. ... *) adding first version of maximum filesize check for the crawler git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2534 6c8d7289-2bf4-0310-a012-ef5d649a1542	2006-09-09 15:06:49 +00:00
auron_x	06b1365066	*) fixed existing protection against divbyzero and removed the new one ... git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2530 6c8d7289-2bf4-0310-a012-ef5d649a1542	2006-09-08 23:43:30 +00:00
orbiter	94d7ced900	fix for last ranking commit ... git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2529 6c8d7289-2bf4-0310-a012-ef5d649a1542	2006-09-08 21:14:57 +00:00
orbiter	cc97a3e9c6	fixed possibly bug with indexOutOfBoundsException ... git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2528 6c8d7289-2bf4-0310-a012-ef5d649a1542	2006-09-08 20:27:45 +00:00
orbiter	03835c2ee8	enhanced search result computation ... git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2527 6c8d7289-2bf4-0310-a012-ef5d649a1542	2006-09-08 20:26:44 +00:00
orbiter	809960ddc6	avoid division by zero ... git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2526 6c8d7289-2bf4-0310-a012-ef5d649a1542	2006-09-08 20:00:19 +00:00
orbiter	ac3419b65f	better debugging for indexOutOfBoundException bug ... git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2525 6c8d7289-2bf4-0310-a012-ef5d649a1542	2006-09-08 19:41:52 +00:00
orbiter	75b03a4580	fix for new ArrayIndexOutOfBoundException ... git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2524 6c8d7289-2bf4-0310-a012-ef5d649a1542	2006-09-08 19:15:55 +00:00
orbiter	a8bc768206	enhancements to ranking evaluation ... git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2523 6c8d7289-2bf4-0310-a012-ef5d649a1542	2006-09-08 16:04:50 +00:00
auron_x	a82e926c5d	*) fix for wrong totalPPM-calculation ... git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2522 6c8d7289-2bf4-0310-a012-ef5d649a1542	2006-09-08 15:19:16 +00:00
theli	33898ae7e9	*) ResourceInfoFactory.java: Bugfix for classNotFoundException ... See: http://www.yacy-forum.de/viewtopic.php?t=2797 git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2521 6c8d7289-2bf4-0310-a012-ef5d649a1542	2006-09-08 13:43:23 +00:00
theli	406e170e25	*) more verbose error message ... git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2519 6c8d7289-2bf4-0310-a012-ef5d649a1542	2006-09-08 11:48:49 +00:00
theli	b298474e22	*) Bugfix needed because of changed plasmaCrawlLURL.load behavior ... git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2518 6c8d7289-2bf4-0310-a012-ef5d649a1542	2006-09-08 11:33:27 +00:00
orbiter	c2e6cc8c6b	small part of Bosts patch ... git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2517 6c8d7289-2bf4-0310-a012-ef5d649a1542	2006-09-08 01:40:23 +00:00
orbiter	96c6e4e322	- enhancements to detailed search page ... - enhancements to search ranking computation process - removed bugs in postranking git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2516 6c8d7289-2bf4-0310-a012-ef5d649a1542	2006-09-08 01:26:06 +00:00
orbiter	9340dbb501	fixed all possible problems with nullpointer exception for LURLs ... git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2513 6c8d7289-2bf4-0310-a012-ef5d649a1542	2006-09-07 18:24:39 +00:00
theli	a5ed86105b	*) bugfix for handling of ResourceInfo object in proxy ... git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2512 6c8d7289-2bf4-0310-a012-ef5d649a1542	2006-09-07 15:50:45 +00:00
hermens	ff4362b02d	some more fixes for new plasmaCrawlLURL.load behavior ... git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2511 6c8d7289-2bf4-0310-a012-ef5d649a1542	2006-09-07 14:32:46 +00:00
hermens	7aeadbe7cc	another NullPointerException in http.ResourceInfo ... git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2510 6c8d7289-2bf4-0310-a012-ef5d649a1542	2006-09-07 13:13:19 +00:00
orbiter	141f9e5bb4	fix for new plasmaCrawlLURL.load behavior ... git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2509 6c8d7289-2bf4-0310-a012-ef5d649a1542	2006-09-07 12:27:32 +00:00
orbiter	1e7fd48afd	added size method to ftpc ... git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2508 6c8d7289-2bf4-0310-a012-ef5d649a1542	2006-09-07 12:21:41 +00:00
hermens	087f7511f8	prevent NullPointerException in http.ResourceInfo ... git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2507 6c8d7289-2bf4-0310-a012-ef5d649a1542	2006-09-07 11:46:22 +00:00
orbiter	a2525072f2	bugfix for kelondroRow - property generation ... this bug affected ranking parameters :-( git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2506 6c8d7289-2bf4-0310-a012-ef5d649a1542	2006-09-07 10:55:34 +00:00
hydrox	59a5511dbb	*) added missing static Strings as requested by theli ... git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2505 6c8d7289-2bf4-0310-a012-ef5d649a1542	2006-09-07 07:20:28 +00:00
theli	6578564c9a	*) Ignore more hop by hop http headers ... git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2504 6c8d7289-2bf4-0310-a012-ef5d649a1542	2006-09-07 05:38:35 +00:00
theli	b44514242a	*) crawler/ftp/CrawlWorker.java: better errorhandling ... git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2503 6c8d7289-2bf4-0310-a012-ef5d649a1542	2006-09-07 05:22:35 +00:00
theli	7d7f30139c	*) crawler/ftp/CrawlWorker.java: delete old cache file ... git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2502 6c8d7289-2bf4-0310-a012-ef5d649a1542	2006-09-07 05:08:35 +00:00
theli	4ae0f122f8	*) ResourceInfo.java: License header added ... git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2501 6c8d7289-2bf4-0310-a012-ef5d649a1542	2006-09-07 05:01:07 +00:00
theli	043edfa4d8	*) ftp/ResourceInfo.java ResourceInfo object for ftp resources added ... *) ftp/CrawlWorker.java better errorhandling for ftp crawler *) plasmaCrawlEURL.java: some errorcodes added git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2499 6c8d7289-2bf4-0310-a012-ef5d649a1542	2006-09-07 04:12:52 +00:00
orbiter	4866868c0e	added write cache for LURLs ... This was necessary to speed up the index receive process during global search git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2498 6c8d7289-2bf4-0310-a012-ef5d649a1542	2006-09-07 01:13:03 +00:00
orbiter	8a0e35618b	enhancements to search result preparation ... - added detailed count on remote search results - enhanced search sequence during remote searches (doing local search in sequence) - strict adherence to timout limits git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2497 6c8d7289-2bf4-0310-a012-ef5d649a1542	2006-09-06 17:51:28 +00:00
theli	5c1bb53d2a	Missing description for last commit ... *) next step of restructuring for new crawlers > HTCaching should now work protocol independent -- introduction of new ResourceInfo objects containing protocolspecific metadata of a resource. -- the ResourceInfo objects now implement old functions like shallIndexCacheForXXX, shallStoreCacheForXXX in a protocol dependent manner > Indexing should also work protocol independent now git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2496 6c8d7289-2bf4-0310-a012-ef5d649a1542	2006-09-06 14:35:45 +00:00
theli	dae763d8e3	git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2495 6c8d7289-2bf4-0310-a012-ef5d649a1542	2006-09-06 14:31:17 +00:00
theli	4825bfaaf3	*) Bugfix for PrintWriter Problem ... See: http://www.yacy-forum.de/viewtopic.php?t=2792 git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2494 6c8d7289-2bf4-0310-a012-ef5d649a1542	2006-09-05 18:55:45 +00:00
orbiter	d4c5e2af01	html-dirlist can now also be generated from existing connections ... git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2493 6c8d7289-2bf4-0310-a012-ef5d649a1542	2006-09-05 10:11:07 +00:00
theli	7930839594	*) URL.java: userinfo was not taken over when generating a new url from a base url and a rel. path ... *) CrawlWorker.java: using new dirhtml function of ftpc git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2492 6c8d7289-2bf4-0310-a012-ef5d649a1542	2006-09-05 05:17:57 +00:00
orbiter	17ba468165	added html dirlisting generation in ftpc.java: ... ftpc.dirhtml() generates a StringBuffer with a complete web page git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2491 6c8d7289-2bf4-0310-a012-ef5d649a1542	2006-09-05 00:11:59 +00:00
theli	7a35b8e237	*) direct access to responseheaders of sbQueue.Entry removed to make it more http independent ... git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2487 6c8d7289-2bf4-0310-a012-ef5d649a1542	2006-09-04 15:36:19 +00:00
theli	ffbf416e76	*) direct access to requestheader of htCache.Entry removed to make it more http independent ... git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2486 6c8d7289-2bf4-0310-a012-ef5d649a1542	2006-09-04 15:29:45 +00:00
theli	3870d615e3	*) setting htCache.Entry fields to private ... git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2485 6c8d7289-2bf4-0310-a012-ef5d649a1542	2006-09-04 15:06:58 +00:00
theli	393a7d10be	*) setting htCache.Entry fields to private ... git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2484 6c8d7289-2bf4-0310-a012-ef5d649a1542	2006-09-04 15:03:54 +00:00
theli	ab5a9bee66	*) adding some copyright headers ... *) next step of restructuring for new crawlers - adding first testversion of ftp crawler class -- does not create a htCache entry yet git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2483 6c8d7289-2bf4-0310-a012-ef5d649a1542	2006-09-04 14:38:29 +00:00
theli	5847492537	*) next step of restructuring for new crawlers ... - IndexCreate_p.java: correcting problems with ftp urls - URL.java does not cutout the userinfo anymore (needed to transport authentication info in ftp urls, e.g. ftp://username:pwd@ftp.irgendwas.de) - plasmaCrawlLoader.java: -- hack to re enable https urls -- adding function getSupportedProtocols git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2482 6c8d7289-2bf4-0310-a012-ef5d649a1542	2006-09-04 13:17:11 +00:00
orbiter	6cce47e217	test of ftp-urls in URL class ... git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2481 6c8d7289-2bf4-0310-a012-ef5d649a1542	2006-09-04 13:10:40 +00:00
theli	fce9e7741b	*) next step of restructuring for new crawlers ... - renaming of http specific crawler settings git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2480 6c8d7289-2bf4-0310-a012-ef5d649a1542	2006-09-04 11:56:47 +00:00
theli	e3f0136606	*) next step of restructuring for new crawlers ... - adding function isSupportedProcotol to plasmaCrawlLoader.java - disabling robots.txt check for protocols other than http(s) git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2479 6c8d7289-2bf4-0310-a012-ef5d649a1542	2006-09-04 11:46:17 +00:00
theli	9ded4e8d5a	*) Bugfix for name resolution in proxy mode ... See: http://www.yacy-forum.de/viewtopic.php?p=25241 git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2478 6c8d7289-2bf4-0310-a012-ef5d649a1542	2006-09-04 11:26:53 +00:00
theli	1c8300fcec	*) Bugfix for name resolution in proxy mode ... See: http://www.yacy-forum.de/viewtopic.php?p=25241 git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2477 6c8d7289-2bf4-0310-a012-ef5d649a1542	2006-09-04 11:23:57 +00:00
theli	4e2a950ac9	*) next step of restructuring for new crawlers ... - avoid using the http crawler class directly. Using the interface class instead git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2476 6c8d7289-2bf4-0310-a012-ef5d649a1542	2006-09-04 09:24:24 +00:00
theli	09b106eb04	*) next step of restructuring for new crawlers ... - adding interface class (plasma/crawler/plasmaCrawlWorker.java) for protocol specific crawl-worker threads - moving reusable code into abstract crawl-worker class AbstractCrawlWorker.java - the load method of the worker threads should not be called directly anymore (e.g. by the snippet fetcher) to crawl a page and wait for the result use function plasmaCrawlLoader.loadSync([...]) git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2474 6c8d7289-2bf4-0310-a012-ef5d649a1542	2006-09-04 09:00:18 +00:00
theli	eb9b138986	*) next step of restructuring for new crawlers ... - conversion of the crawler pool into a keyed object pool - crawlers are now loaded based on the url protocol (of course works only for http now) git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2473 6c8d7289-2bf4-0310-a012-ef5d649a1542	2006-09-04 06:52:55 +00:00
theli	1395aae742	*) starting restructuring which is needed to add crawlers for additional protocols ... git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2472 6c8d7289-2bf4-0310-a012-ef5d649a1542	2006-09-04 06:09:20 +00:00
theli	b4acbdaa97	*) better handling of server shutdown ... See: e.g. http://www.yacy-forum.de/viewtopic.php?p=25234 git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2470 6c8d7289-2bf4-0310-a012-ef5d649a1542	2006-09-04 05:17:37 +00:00
theli	f3ac4dbbb9	*) better handling of server shutdown ... See: e.g. http://www.yacy-forum.de/viewtopic.php?t=2584 git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2468 6c8d7289-2bf4-0310-a012-ef5d649a1542	2006-09-03 14:59:00 +00:00
theli	959b779aba	*) avoid performance loss if log level is greater than 'fine' ... See: http://www.yacy-forum.de/viewtopic.php?p=25180 git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2467 6c8d7289-2bf4-0310-a012-ef5d649a1542	2006-09-03 08:42:46 +00:00
auron_x	57dda1a92c	*)again fixing for wrong version display, now totally working with double instead of float ... git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2464 6c8d7289-2bf4-0310-a012-ef5d649a1542	2006-08-28 17:54:07 +00:00
auron_x	479b74e1dd	*) fix for stupid mistake in new ppm-calc which caused decimal digits beeing written to seedinfo ... git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2463 6c8d7289-2bf4-0310-a012-ef5d649a1542	2006-08-28 04:43:28 +00:00
auron_x	348258a557	*) changed PPM-calculation to be much more accurate ... git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2461 6c8d7289-2bf4-0310-a012-ef5d649a1542	2006-08-27 17:18:34 +00:00
orbiter	18b6876860	new cache flush configuration settings ... git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2460 6c8d7289-2bf4-0310-a012-ef5d649a1542	2006-08-25 22:31:21 +00:00
hermens	f0278b4092	Bugfix for / by zero when the AssortmentCluster is empty ... See: http://www.yacy-forum.de/viewtopic.php?t=2746 git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2459 6c8d7289-2bf4-0310-a012-ef5d649a1542	2006-08-25 20:23:04 +00:00
orbiter	14e0bb0dcf	allow more references per word for new db ... git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2458 6c8d7289-2bf4-0310-a012-ef5d649a1542	2006-08-25 12:06:23 +00:00
orbiter	985dcbde7f	changed some parameters that may cause better memory usage and more indexing speed ... git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2457 6c8d7289-2bf4-0310-a012-ef5d649a1542	2006-08-24 23:39:52 +00:00
orbiter	b7f4a1521b	added options to switch on or off the kelondroFlexTable for NURL, EURL and PreNURL ... git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2456 6c8d7289-2bf4-0310-a012-ef5d649a1542	2006-08-24 22:21:22 +00:00
orbiter	c26da4893b	turned back NURL usage of kelondroTree, kelondroFlexTable has still problems with deleted entries ... git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2454 6c8d7289-2bf4-0310-a012-ef5d649a1542	2006-08-24 10:03:38 +00:00
orbiter	db1eae0227	* simplified initialization of database objects ... * replaced kelondroTree for NURLs by kelondroFlex * replaced kelondroTree for EURLs by kelondroFlex take care, may be very buggy please finish crawls before updating. crawls will be lost. git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2452 6c8d7289-2bf4-0310-a012-ef5d649a1542	2006-08-24 02:19:25 +00:00
hermens	0b73f2b132	Repair DNS prefetch during cacheScan ... git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2451 6c8d7289-2bf4-0310-a012-ef5d649a1542	2006-08-24 01:31:08 +00:00
orbiter	27a159b401	* documentation update ... * removed doc from release * release information in doc/News.html * release 0.46 git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2442 6c8d7289-2bf4-0310-a012-ef5d649a1542	2006-08-23 11:36:09 +00:00
theli	f80f776b89	*) Trying to solve NullpointerException problem in function addURLtoErrorDB ... See: http://www.yacy-forum.de/viewtopic.php?t=2705 git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2441 6c8d7289-2bf4-0310-a012-ef5d649a1542	2006-08-23 10:23:20 +00:00
orbiter	d78b824e85	fixed problem with default path after first start-up ... git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2440 6c8d7289-2bf4-0310-a012-ef5d649a1542	2006-08-22 13:35:51 +00:00
hydrox	1c99b5a484	*)fixed logging for urldbcleanup ... *)changed exception handling in urldbcleanup so that it shows NullPointerException correctly *)added more Blacklisting to urlcleaner git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2436 6c8d7289-2bf4-0310-a012-ef5d649a1542	2006-08-21 06:42:42 +00:00
orbiter	135e019883	removed one superfluous line from last commit ... (hasnot is included in remove) git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2435 6c8d7289-2bf4-0310-a012-ef5d649a1542	2006-08-21 01:59:44 +00:00
orbiter	1591a55963	added object cache miss-cache use for remove method ... git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2434 6c8d7289-2bf4-0310-a012-ef5d649a1542	2006-08-21 01:51:27 +00:00
orbiter	8f3f4ab0eb	enhanced synchronisation in plasmaWordIndex ... git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2433 6c8d7289-2bf4-0310-a012-ef5d649a1542	2006-08-21 01:29:26 +00:00
orbiter	f933f00f09	another patch to URL protocol handling for 'news', 'nntp' etc: ... reject it! (the java.net.URL class rejects them too) git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2432 6c8d7289-2bf4-0310-a012-ef5d649a1542	2006-08-21 01:04:04 +00:00
orbiter	4c6e00d80a	more bugfixes for URL class, see: ... http://www.yacy-forum.de/viewtopic.php?p=24844#24844 git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2431 6c8d7289-2bf4-0310-a012-ef5d649a1542	2006-08-21 00:23:39 +00:00
orbiter	23dd972608	fixed memory calculation in performanceMemory web page ... fixed also maximum cache size computation git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2429 6c8d7289-2bf4-0310-a012-ef5d649a1542	2006-08-20 01:20:34 +00:00
orbiter	b7dc251948	fixed bugs in url class: ... - correct backpath ('..') handling - correct absolute path handling - included https git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2428 6c8d7289-2bf4-0310-a012-ef5d649a1542	2006-08-19 22:27:01 +00:00
orbiter	1ce3c22761	better memory control: ... - added memory monitor for preNURL-db in performanceMemory - changed default memory assignments git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2427 6c8d7289-2bf4-0310-a012-ef5d649a1542	2006-08-19 13:09:04 +00:00
orbiter	39b4c26bdc	more memory control: ... - catchup of OutOfMemoryError in server threads - automatic adoption of word cache size after a Short Mem Cycle git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2426 6c8d7289-2bf4-0310-a012-ef5d649a1542	2006-08-19 00:06:39 +00:00
orbiter	3e9d509c39	some small fixes ... git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2425 6c8d7289-2bf4-0310-a012-ef5d649a1542	2006-08-18 22:50:05 +00:00
orbiter	276225d79e	fix for URL class ... git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2423 6c8d7289-2bf4-0310-a012-ef5d649a1542	2006-08-18 21:33:00 +00:00
orbiter	eb633c0a4f	server threads must now supply a method that can be called in case ... of short memory. This has been realized for the indexing thread. git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2421 6c8d7289-2bf4-0310-a012-ef5d649a1542	2006-08-18 02:07:03 +00:00
orbiter	f5720cb2fa	removed most synchronization in wordIndex (for testing) ... git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2420 6c8d7289-2bf4-0310-a012-ef5d649a1542	2006-08-18 01:35:33 +00:00
orbiter	0187c60010	because of a bug in the JRE 1.4.2 there was no memory protection ... see http://bugs.sun.com/bugdatabase/view_bug.do?bug_id=4686462 this commit fixes the bug by using a memory-computation patch. All uses of Runtime.maxMemory had been replaced by serverMemory.max The bug is not present any more in Java 1.5 git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2419 6c8d7289-2bf4-0310-a012-ef5d649a1542	2006-08-18 01:33:54 +00:00
auron_x	4eca0f8830	*) fixed PPM calculation for multiple indexer-threads ... *) fixed totalPPM calculation and added total PPM to Network.html git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2418 6c8d7289-2bf4-0310-a012-ef5d649a1542	2006-08-17 19:15:30 +00:00
orbiter	cfb51fdef1	less synchronization in plasmaWordIndex ... git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2416 6c8d7289-2bf4-0310-a012-ef5d649a1542	2006-08-17 00:10:50 +00:00
orbiter	d6a928c2da	quickfix for http://www.yacy-forum.de/viewtopic.php?t=2705 ... git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2415 6c8d7289-2bf4-0310-a012-ef5d649a1542	2006-08-16 23:20:21 +00:00
orbiter	6ad471ef96	* applied many compiler warning recommendations ... * cleaned up code * added unit test code * migrated ranking RCI computation to kelondroFlex and kelondroCollectionIndex git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2414 6c8d7289-2bf4-0310-a012-ef5d649a1542	2006-08-16 19:49:31 +00:00
allo	cf1186597b	utf fix from theli ... git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2412 6c8d7289-2bf4-0310-a012-ef5d649a1542	2006-08-16 15:26:04 +00:00
hydrox	9da3aa74d3	silly me, fix for the fix as advised by theli ... git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2408 6c8d7289-2bf4-0310-a012-ef5d649a1542	2006-08-13 17:26:32 +00:00
hydrox	bb3d9a5582	*) e.getMessage().indexOf() can only be used if there is actually an ExceptionMessage. ... git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2407 6c8d7289-2bf4-0310-a012-ef5d649a1542	2006-08-13 17:18:09 +00:00
hydrox	7a54010a9c	*) Iterators can't be casted to IndexContainer ... git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2406 6c8d7289-2bf4-0310-a012-ef5d649a1542	2006-08-13 17:08:39 +00:00
theli	5e0b6f8f83	*) sorting peer name list on Blacklist_p.html ... *) restructuring of sharedBlacklist_p.java git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2405 6c8d7289-2bf4-0310-a012-ef5d649a1542	2006-08-13 13:29:50 +00:00
orbiter	cd5f7e137c	fixed problem with NURL-generation upon first startup ... (a new kelondroFlexTable was generated, which should not) git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2402 6c8d7289-2bf4-0310-a012-ef5d649a1542	2006-08-12 23:24:50 +00:00
orbiter	8418af141a	added several consistency checks and small changes ... git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2400 6c8d7289-2bf4-0310-a012-ef5d649a1542	2006-08-12 15:59:14 +00:00
theli	9d13aeca13	*) removing class. does not work so far ... git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2399 6c8d7289-2bf4-0310-a012-ef5d649a1542	2006-08-12 14:43:07 +00:00
theli	95a84ae469	*) adding missing classes ... git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2398 6c8d7289-2bf4-0310-a012-ef5d649a1542	2006-08-12 14:41:26 +00:00
theli	eee44be602	*) adding an interface for customized blacklist classes ... - now it's possible to use a customized blacklist engine instead of the default one - this can be done by configuring the property BlackLists.class See: http://www.yacy-forum.de/viewtopic.php?t=2108 git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2397 6c8d7289-2bf4-0310-a012-ef5d649a1542	2006-08-12 14:28:14 +00:00
orbiter	6d2f15971a	there is a very strange error that causes that the kelondroRecords structure ... is corrupted. The cause is, that the deleted-records-chain has wrong entries, and one of the pointers in that chain points to a place behind the file end. This causes an IndexOutOfBoundsException within an IO operation. I currently don't know the reason that the deleted-records-chain is corrupted, but the error can be catched. If this now happens with the assortment database, the database is deleted. See also: http://www.yacy-forum.de/viewtopic.php?p=24586#24586 git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2396 6c8d7289-2bf4-0310-a012-ef5d649a1542	2006-08-12 13:45:23 +00:00
theli	d2e8e76218	*) now it's possible to configure the yacy blacklist separately for dht, search, proxy, crawler ... See: http://www.yacy-forum.de/viewtopic.php?t=2541 http://www.yacy-forum.de/viewtopic.php?p=24516 git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2389 6c8d7289-2bf4-0310-a012-ef5d649a1542	2006-08-12 02:42:10 +00:00
orbiter	9ae9062bd3	* disabled new kelondroFlex table for NURLs ... * added new RAM index Class * fixed possible synchronization problem in kelondroRecords git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2388 6c8d7289-2bf4-0310-a012-ef5d649a1542	2006-08-12 00:58:43 +00:00
orbiter	689bbcf9cd	replaced kelondroTree db for NURLs by new kelondroFlexTable ... The new database is only created if the old is deleted or does not exist git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2387 6c8d7289-2bf4-0310-a012-ef5d649a1542	2006-08-11 23:36:58 +00:00
orbiter	7fbba41962	synchronization fixes ... git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2386 6c8d7289-2bf4-0310-a012-ef5d649a1542	2006-08-11 23:04:36 +00:00
orbiter	328f9859a5	more synchronization in plasmaWordIndex ... git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2385 6c8d7289-2bf4-0310-a012-ef5d649a1542	2006-08-11 22:07:59 +00:00
orbiter	f43c90fa98	fixed handling of null referer in crawlOrder ... git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2384 6c8d7289-2bf4-0310-a012-ef5d649a1542	2006-08-11 21:46:34 +00:00
orbiter	130e6d4719	generalized index object for eurl, nurl and lurl to prepare move ... of these tables to new kelondroFlexTable Object git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2382 6c8d7289-2bf4-0310-a012-ef5d649a1542	2006-08-11 17:37:54 +00:00
orbiter	acdf24877f	more synchronization against outOfMemoryError in wordIndex ... git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2381 6c8d7289-2bf4-0310-a012-ef5d649a1542	2006-08-11 16:27:56 +00:00
orbiter	95160d7f2c	fixed size computation of index elements from the collection index ... git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2380 6c8d7289-2bf4-0310-a012-ef5d649a1542	2006-08-11 16:01:18 +00:00
orbiter	26116cabde	added missing rowdef assignment ... git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2379 6c8d7289-2bf4-0310-a012-ef5d649a1542	2006-08-11 15:31:40 +00:00
orbiter	cfbacbbf08	reverted change in robotsParser ... git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2378 6c8d7289-2bf4-0310-a012-ef5d649a1542	2006-08-11 15:29:29 +00:00
orbiter	abf22f6e60	removed url normalform computation from htmlFilterContentScraper. ... This method was implemented in de.anomic.net.URL git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2377 6c8d7289-2bf4-0310-a012-ef5d649a1542	2006-08-11 15:09:22 +00:00
orbiter	740d49751d	* strict type and size check in kelondroRow handling ... * adopted all code to use the declaration form of kelondroRow * fixed a bug in kelondroRow which caused wrong parsing of encoding type * the bug caused bad database behaviour in new indexCollection data structure. because of this bug, all test databases are now already void. A new database is created * the kelondroFlexTable and indexCollection data structures now store a declaration of the row definition into a properties file along the database files. git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2375 6c8d7289-2bf4-0310-a012-ef5d649a1542	2006-08-11 03:20:44 +00:00
orbiter	314021453f	* more logging ... * option in yacy.init to set useCollectionIndex usage git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2374 6c8d7289-2bf4-0310-a012-ef5d649a1542	2006-08-10 21:21:50 +00:00
allo	a52f36787f	better templatedebugging ... git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2371 6c8d7289-2bf4-0310-a012-ef5d649a1542	2006-08-10 14:02:03 +00:00