Marown/yacy_search_server
1
0
Fork 0
mirror of https://github.com/yacy/yacy_search_server.git synced 2024-09-21 00:00:13 +02:00
Code Issues Actions Packages Projects Releases Wiki Activity
1,463 Commits 1 Branch 20 Tags 836 MiB
50d85657b8
Commit Graph

510 Commits

Author SHA1 Message Date
theli
6adf8a4bde *) Renaming Logger function names to reflect the proper Java Logging API Loglevels 
- please use logFine instead of logDebug
   - please use logFailure instead of logError
   See: http://www.yacy-forum.de/viewtopic.php?p=8726#8726

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@614 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2005-08-30 21:10:39 +00:00
theli
f19c09b227 *) Suppress stacktrace on crawler error for "MalformedURLException" 
See: http://www.yacy-forum.de/viewtopic.php?p=8733#8733

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@613 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2005-08-30 20:25:07 +00:00
theli
cc1df08069 *) Adding missing synchronized blocks 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@608 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2005-08-30 14:57:32 +00:00
borg-0300
bf14e6def5 *) proxyCache, proxyCacheSize can be changed under 'Proxy Indexing' 
- path now are absolute
*) move path check from plasmaHTCache to plasmaSwitchboard
   - only one path check when starting
*) small other

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@606 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2005-08-30 12:50:30 +00:00
theli
9b818b1ce3 *) Pausing Crawlers if there is not enough space on disk 
See: http://www.yacy-forum.de/viewtopic.php?p=8648

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@603 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2005-08-30 09:43:27 +00:00
theli
b33094e925 *) Trying to solve "Too many open files bug" 
*) Temp.Bugfix for "Bug in Index Restore"
   See: http://www.yacy-forum.de/viewtopic.php?p=8647#8647
   Orbiter: Please take a look



git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@602 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2005-08-30 09:07:42 +00:00
theli
34790acf02 *) Bugfix for suppressing of stacktrace in log on crawler error "unknown host" 
See: http://www.yacy-forum.de/viewtopic.php?p=8615#8615

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@600 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2005-08-30 06:24:23 +00:00
theli
af7b8f75bd *) Making proxyAccessLogging configureable via yacy.logging file 
- logging can be disabled now
   - logging directory / filelimit / rotation count can be configured now
   See: http://www.yacy-forum.de/viewtopic.php?t=965&postdays=0&postorder=asc&start=30#8280

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@595 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2005-08-29 11:31:58 +00:00
theli
2a081c9ee5 *) Adding additional logging message for "NURL.entry() == null" Bug 
See: http://www.yacy-forum.de/viewtopic.php?p=8446

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@591 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2005-08-28 05:39:26 +00:00
theli
cb1f11c96b *) Suppress stacktrace on crawler error for "Unknown Host" 
See: http://www.yacy-forum.de/viewtopic.php?p=8431

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@590 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2005-08-28 05:08:26 +00:00
theli
e338a13de3 *) Suppress stacktrace on crawler error for "Read timed out" 
See: http://www.yacy-forum.de/viewtopic.php?p=8433

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@589 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2005-08-27 18:48:02 +00:00
theli
2e43e744de *) Suppress stacktrace on crawler error for "connect timed out" 
See: http://www.yacy-forum.de/viewtopic.php?p=8420 

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@588 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2005-08-27 04:53:25 +00:00
theli
36cbe04e3e *) Bugfix for Crawler Redirection Bug 
See: http://www.yacy-forum.de/viewtopic.php?p=8422

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@587 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2005-08-27 04:36:13 +00:00
theli
b70de495a0 *) Remembering Crawler-isPaused setting 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@586 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2005-08-25 09:51:24 +00:00
theli
e569a84dc0 *) Using the same configuration settings for all indexing threads on server Startup 
See: http://www.yacy-forum.de/viewtopic.php?p=8349

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@584 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2005-08-24 09:34:04 +00:00
theli
17be77a468 *) Bugfix for "Crawler data will not be removed from htcache if content parsing failed" 
See: http://www.yacy-forum.de/viewtopic.php?t=965&highlight=ramdisk
*) Making ACCEPT_LANGUAGE configureable for crawler
   See: http://www.yacy-forum.de/viewtopic.php?p=8327

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@583 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2005-08-24 07:47:42 +00:00
theli
5f55dff297 *) Bugfix for "Binäre Nullen auf der page: Index Creation: Indexing Queue" 
See: http://www.yacy-forum.de/viewtopic.php?p=6877#6877

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@577 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2005-08-23 08:37:42 +00:00
allo
eb6365c069 local Bootstrapping bug. 
use yacyDebugMode=true to allow local bootstrapping


git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@572 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2005-08-22 12:13:19 +00:00
theli
330eae7cf3 *) Normalizing CrawlerStartURL now before crawling is started 
*) CrawlWorker also does a URL normalization now before following the redirection URL
*) CrawlWorker removes redirection URL correctly from noticeURL stack now

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@571 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2005-08-21 22:52:46 +00:00
theli
ab894d26bc *) Bugfix for "plasmaSwitchboard.deQueue: null" Bug (hopefully) 
See: http://www.yacy-forum.de/viewtopic.php?p=8135#8135

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@570 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2005-08-21 22:48:37 +00:00
theli
eaf9f26cc3 *) Bugfix for NULL PROFILE HANDLE 'null' Bug: 
See: http://www.yacy-forum.de/viewtopic.php?p=7855#7855

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@569 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2005-08-21 22:16:19 +00:00
rramthun
4cb382decb Adding changes by borg-0300 from http://www.yacy-forum.de/viewtopic.php?t=997 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@565 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2005-08-20 17:05:01 +00:00
theli
ec4c70d722 *) If there are at most 10 entries left while doing an index transfer, these entries will also be appended 
to the index list
   |> D 2005/08/18 10:00:02 PLASMA Selected partial index (33 from 37 URLs, 0 not bound) for word fSuQM0xAJK1G
   See: http://www.yacy-forum.de/viewtopic.php?t=970

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@556 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2005-08-18 10:04:45 +00:00
theli
d4a045d7b1 *) Trying to solve "de.anomic.plasma.plasmaSwitchboard.deQueue': null" Bug 
See: http://www.yacy-forum.de/viewtopic.php?p=7791

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@555 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2005-08-18 06:40:26 +00:00
theli
ea9a992f05 *) Before the crawler retries to download a URL it checks if the server is already doing a shutdown 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@554 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2005-08-17 11:36:48 +00:00
theli
ea26b84eed *) Bugfix for http://www.yacy-forum.de/viewtopic.php?t=954 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@553 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2005-08-17 10:12:52 +00:00
theli
0c8a48e2cb *) converting php Session ID to lower case in funktion isCGI 
See: http://www.yacy-forum.de/viewtopic.php?p=7671#7671

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@552 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2005-08-17 05:50:18 +00:00
orbiter
e616395c3b latest changes and cut for 0.40 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@548 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2005-08-16 15:15:19 +00:00
orbiter
c47bb1182d bugfix for assortment initialization error 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@547 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2005-08-16 11:43:14 +00:00
theli
4654eae4e2 *) adding php Session ID to argument in funktion isCGI 
See: http://www.yacy-forum.de/viewtopic.php?p=7671#7671

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@546 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2005-08-16 11:33:31 +00:00
orbiter
25f632dbd9 more DHT bugfixes and better logging of DHT effects 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@542 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2005-08-16 00:31:15 +00:00
orbiter
5cb00889d9 enhancements to dht selection, search and search presentation 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@540 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2005-08-15 01:12:25 +00:00
orbiter
ba0a486328 moved printStackTrace() to logging 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@539 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2005-08-14 23:35:18 +00:00
orbiter
3094045d34 fix for http://www.yacy-forum.de/viewtopic.php?p=7454#7454 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@536 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2005-08-14 16:16:11 +00:00
orbiter
cd10370992 several bugfixes and dht selection / logging improvement 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@531 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2005-08-14 00:57:30 +00:00
orbiter
3610fe6b3a see http://www.yacy-forum.de/viewtopic.php?p=7410#7410 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@530 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2005-08-13 22:04:18 +00:00
orbiter
c8a7a85ce2 fix for http://www.yacy-forum.de/viewtopic.php?p=7384#7384 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@529 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2005-08-13 21:41:22 +00:00
orbiter
6594541ef5 fix for http://www.yacy-forum.de/viewtopic.php?p=7361#7361 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@526 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2005-08-12 22:53:49 +00:00
orbiter
7db543a9fa fixes for several dht misbehaviours 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@524 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2005-08-12 22:14:24 +00:00
orbiter
5716f8521d bug fixes for word ordering and dht index selection 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@521 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2005-08-12 14:06:47 +00:00
orbiter
f5259f29e8 word cache behaviour fix and other fixes 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@519 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2005-08-11 23:33:19 +00:00
orbiter
2c234e1b82 better log output for search result 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@512 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2005-08-10 09:52:50 +00:00
theli
89c9faa89e *) More graceful logging output in crawler 
See: http://www.yacy-forum.de/viewtopic.php?t=894

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@511 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2005-08-10 06:15:47 +00:00
orbiter
248c24b60a intermission-feature usage in case of local and remote search 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@510 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2005-08-09 20:43:37 +00:00
theli
b32e7c516c git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@507 6c8d7289-2bf4-0310-a012-ef5d649a1542 2005-08-09 09:07:19 +00:00
theli
86305f051d *) Trying to solve "java.net.BindException: Address already in use: JVM_Bind" Problem 
by retrying Socket bind
   See: http://www.yacy-forum.de/viewtopic.php?p=6935

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@497 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2005-08-06 14:38:58 +00:00
theli
865b9490a2 *) Making DHT Transfer while Crawling configurable 
See: http://www.yacy-forum.de/viewtopic.php?p=6904

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@496 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2005-08-06 11:40:12 +00:00
theli
1d83d7e4d7 *) httpdFileHandler.java: 
no stacktrace will be printed into log file for "Connection timed out" Errors now
   See: http://www.yacy-forum.de/viewtopic.php?p=6381

*) plasmaCrawlWorker.java:
   If a "Read timed out" error occurs while crawling a site, the failed crawl will be
   retried.

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@493 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2005-08-04 11:05:04 +00:00
orbiter
2d8557cb10 minor changes 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@487 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2005-08-03 02:02:39 +00:00
orbiter
91163db52e fix for more time-related problems in proxy 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@486 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2005-08-03 00:52:32 +00:00
orbiter
fb6f238d70 fix for expires-problem 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@485 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2005-08-03 00:28:12 +00:00
rramthun
eacff63eda Typos... 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@482 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2005-08-02 16:09:19 +00:00
orbiter
40da910f41 bugfixes and automatic news-cleanup 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@481 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2005-08-02 16:03:35 +00:00
theli
228b04b499 *) Bugfix for "wrong seed-upload timestamp" problem 
http://www.yacy-forum.de/viewtopic.php?t=817

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@480 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2005-08-02 15:36:10 +00:00
theli
470839a16a *) Crawler/Session pool settings will now be stored properly into configfile 
Bugfix for:
- http://www.yacy-forum.de/viewtopic.php?t=502
- http://www.yacy-forum.de/viewtopic.php?t=778

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@477 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2005-08-02 12:20:03 +00:00
orbiter
4377e119f3 bugfix for http://www.yacy-forum.de/viewtopic.php?p=6620#6620 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@476 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2005-08-02 10:41:59 +00:00
orbiter
e84a177c49 many bigfixes 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@475 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2005-08-02 02:18:01 +00:00
orbiter
7e3e9ba0de fix for http://www.yacy-forum.de/viewtopic.php?p=6563#6563 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@472 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2005-08-01 20:36:27 +00:00
orbiter
1022fbeb65 many YaCyNews fixes 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@461 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2005-07-31 01:54:46 +00:00
orbiter
13abd8b6e7 added news-creation at crawl start 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@460 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2005-07-30 11:57:19 +00:00
rramthun
f555b9d5f2 Translation, spelling... 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@459 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2005-07-30 11:11:32 +00:00
orbiter
cdbbfd50fb fixed bad remote crawl behavior 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@453 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2005-07-29 06:50:36 +00:00
orbiter
36707586c7 filtering of jsessionid 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@447 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2005-07-27 23:20:17 +00:00
rramthun
6f2f54a312 Translation, spelling... 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@444 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2005-07-27 20:29:35 +00:00
orbiter
81e564edb8 faster crawl profile list cleanup 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@442 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2005-07-27 14:16:47 +00:00
orbiter
ad90f0ad13 activated RWI distribution to DHT for senior peers (default redundancy 3), necessary now for network growth 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@438 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2005-07-27 12:51:00 +00:00
orbiter
b9d18d40cb configuration of proxy idle time in performance menue 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@436 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2005-07-26 15:17:29 +00:00
orbiter
3470a72d48 fixed div by zero, set default delays, fixed release number format and display 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@435 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2005-07-26 11:47:50 +00:00
orbiter
be1f324fca performance setting for remote indexing configuration and latest changes for 0.39 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@424 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2005-07-22 13:56:19 +00:00
orbiter
c64970fa47 re-implemented proxy-busy-check and fixed some other things 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@421 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2005-07-21 11:17:04 +00:00
orbiter
b73557ed2d better assortment monitoring and enhanced profile menue 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@416 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2005-07-20 13:03:41 +00:00
orbiter
1f36bf4dae enhanced assortment capacity; added extended WORDS migration 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@412 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2005-07-20 00:39:06 +00:00
rramthun
0f11399d16 Some corrections... 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@409 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2005-07-19 16:07:13 +00:00
orbiter
9f505af7aa preparations for bulk remote crawls 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@408 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2005-07-19 00:26:31 +00:00
orbiter
9c72b4cdec replaced index dump stack by an dump array and limited url number in assortment ram (prevents too much RAM occupation) 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@406 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2005-07-18 13:32:44 +00:00
orbiter
51962d55bf added 'PPM', page-per-minute statistics 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@405 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2005-07-18 00:44:51 +00:00
orbiter
159f795f65 bugfix (null pointer exception in assortments) 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@404 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2005-07-17 22:25:50 +00:00
orbiter
1d2155675b changed assortment memory cache flush 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@403 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2005-07-17 21:22:18 +00:00
orbiter
19dbed7cc8 code clean-up 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@401 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2005-07-12 15:09:35 +00:00
orbiter
00f63ea00d fail-save patch for pattern matching 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@400 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2005-07-12 00:31:35 +00:00
orbiter
0a6be961ea added pattern organization 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@399 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2005-07-12 00:21:46 +00:00
orbiter
40036ba69c fixed dht transmission; added url-blacklist blocking also for remote search 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@398 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2005-07-12 00:07:09 +00:00
orbiter
311e627363 blocking of blacklisted urls in indexReceive and small changes 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@397 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2005-07-11 15:36:10 +00:00
orbiter
2f0d7ea8d3 removed htcache stati (superfluous now) 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@396 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2005-07-09 00:33:34 +00:00
orbiter
277048501e bugfix 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@395 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2005-07-08 16:24:07 +00:00
orbiter
8b89c46afe fixed problem with cache write 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@394 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2005-07-08 16:06:35 +00:00
orbiter
455ae9f55f fixed htcache-store problem and due-time for remote crawls 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@393 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2005-07-08 15:17:50 +00:00
theli
55d10b864c *) further improvements in shutdown behaviour 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@392 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2005-07-07 22:03:52 +00:00
orbiter
419f8fb398 fixed bugs/missing code regarding new crawl stack 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@384 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2005-07-07 01:38:49 +00:00
orbiter
112c5d3332 the new file-based indexing queue 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@382 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2005-07-06 14:50:01 +00:00
orbiter
858cd94299 replaced indexing ram-queue by file-based stack-queue 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@381 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2005-07-06 14:48:41 +00:00
theli
57c30f1d78 *) bugfix for usage of httpc without gzip content encoding 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@369 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2005-07-04 11:25:25 +00:00
theli
0e2c33ee55 *) Network.html/Network.java: 
- Adding function to manually force peer ping to remote yacy peer
  See:Network.html?page=4
- for debugging purpose only!

*) serverAbstractThread.java:
- Adding posibility to notify a server thread via a synchronization object
- this is needed e.g. by the port forwarding feature to send a notification
  to the peerPing thread to redo peer-ping with the new ip/port Settings_p.html

*) Port Forwarding Feature (it should work now)
- adding a serverThread which is responsible to detect broken port forwarding 
  connections and to do reconnect if needed
- serverCore.java: moving port forwarding initialization into a separate function
- adding positility to configure the ssh port 
- moving configuration section on the gui into a separate fieldset
- hello.java: only trying to do a second connect to the clientIp address during
  peer handshake if either remote port forwarding is not enabled locally or
  the clientIP is not equal to any local ip

*) httpdFileHandler.java:
- printout a more verbose errormessage

*) httpc.java
- allowing to deactivate content encoding from outside


 

*) plasmaCrawlWorker.java
- the crawler worker now tries to refetch the content of a website without
  gzip content encoding if a gzip error occured



git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@368 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2005-07-04 11:09:48 +00:00
orbiter
5159a090b0 fixed parser bug with lowercase force (appeared in: http://spellbound.sourceforge.net/) 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@367 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2005-07-03 23:33:25 +00:00
orbiter
7f7cbc5019 fixed bug with snippets 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@365 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2005-07-03 13:37:20 +00:00
orbiter
eb74fa0c82 fixed a bug with snippet-length 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@359 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2005-07-01 23:35:36 +00:00
orbiter
86f2aa8478 fixed seed-load date bug (evaluating server date for age computation) 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@354 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2005-06-30 23:19:08 +00:00
orbiter
664bceced5 removed debug-lines 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@351 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2005-06-30 18:56:01 +00:00
orbiter
75ebdbc852 enhanced snippet-generation (case where snippet is too long) 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@350 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2005-06-30 18:54:00 +00:00
orbiter
8a4f297324 fixed/enhanced snippet error-handling; suppression of results where no snippet exists 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@347 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2005-06-30 00:01:53 +00:00
orbiter
712fe9ef18 bugfixed utf-8 decoding and parser 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@346 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2005-06-29 22:55:37 +00:00
theli
eee6322aaf *) Adding redirection support to plasmaCrawlWorker.java 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@328 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2005-06-28 08:07:41 +00:00
theli
cd279907c0 *) Adding redirection support to plasmaCrawlWorker.java 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@327 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2005-06-28 08:01:26 +00:00
theli
6697d5e52e *) correcting fkt. mediaExtContains 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@326 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2005-06-28 06:44:31 +00:00
orbiter
3addf58046 enhanced snippet-loading with threads 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@322 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2005-06-24 07:41:07 +00:00
orbiter
56d28a16f0 bugfixes 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@320 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2005-06-23 14:40:39 +00:00
orbiter
d6c85228a6 enhanced snippet computation 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@319 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2005-06-23 12:12:12 +00:00
theli
fafda068f9 *) allowing crawler to process resources with statuscode 203 
- this is needed if yacy is behind a second proxy 

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@316 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2005-06-23 10:00:31 +00:00
theli
aae9a433a6 *) correcting usage of supportedFileExt-List 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@315 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2005-06-23 07:43:59 +00:00
orbiter
1e7f062350 many bugfixes, memory leak fixes, performance enhancements; new kelondroHashtable; activated snippets 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@313 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2005-06-23 02:07:45 +00:00
orbiter
68dc2b0c6b added kelondroArray, the basis for upcoming kelondroHash and some bug fixes 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@311 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2005-06-21 01:17:25 +00:00
orbiter
a19541e563 code-enhancements after analysis with AppPerfect 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@307 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2005-06-20 16:36:31 +00:00
orbiter
85075269a6 extended fail-safe memory-managament. prevents too much allocation, too often GC and should help for the 100%CPU-bug 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@303 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2005-06-20 00:46:23 +00:00
orbiter
e3c92818db avoiding OutOfMemoryError routines 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@302 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2005-06-19 13:37:17 +00:00
orbiter
3e8ee5a46d enhanced caching in kelondroRecords and added better synchronization/finalizer 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@301 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2005-06-19 05:27:42 +00:00
theli
db3ed75728 *) closing stream correctly 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@293 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2005-06-17 07:58:02 +00:00
orbiter
5d06ded005 enhanced html parser speed 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@290 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2005-06-17 01:26:51 +00:00
orbiter
5a490aa065 fixed html parser 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@289 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2005-06-16 21:49:56 +00:00
orbiter
a25b5b4986 fixed possible memory leak in htmlScraper: be aware that now links can get lost; further work necessary 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@288 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2005-06-16 18:31:28 +00:00
theli
9e47ba5ad6 *) adding missing calls for function close() to avoid "too many open file" bug 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@282 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2005-06-16 08:34:52 +00:00
theli
9a98988c3c *) Bugfix for SSL/NIO Bug 
See: http://www.yacy-forum.de/viewtopic.php?t=516
   - removing NIO from server/serverCore.java because of massive problems
     with socket close issues
*) Adding support for remote port forwarding via sch
   @Orbiter: Please take a look into
   - hello.java
   - server/serverCore.java.publicIP()
   - yacy/yacyClient.java.publishMySeed(...)
*) Making startup loading of additional content parsers more failsafe


git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@281 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2005-06-16 07:28:07 +00:00
orbiter
a1ffc27041 preparations for image/movie/music indexing 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@280 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2005-06-16 00:31:13 +00:00
orbiter
a5b40923b6 added word migration to assortments (start with 'java -classpath classes yacy -migratewords') 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@278 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2005-06-15 01:22:07 +00:00
theli
890e3f4d4a *) adding missing calls for function close() to avoid "too many open file" bug*) adding 
*) bugfix in plasma/plasmaParser.java:
   - parsers with missing dependencies wehre not ignored correctly
*) passing a logger instance to the parsers modules which can be used 
   for logging purposes by the parsers (not done yet)


git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@276 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2005-06-13 13:49:17 +00:00
theli
6dd3ec0dc4 *) Adding debug="true" debuglevel="lines,vars,source" to ant build files 
See: http://www.yacy-forum.de/viewtopic.php?p=4099


git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@270 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2005-06-12 05:22:06 +00:00
orbiter
4f9c30ef49 using mime-type instead of file extension for doctype 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@269 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2005-06-10 12:34:18 +00:00
theli
ee9e110366 *) removing old logging configuration properties from yacy.init 
*) serverLog.java logging functions now also accept exceptions als
   additional parameters.
   The Stacktrace of this ecceptions will then be appended to the 
   logging message and can e.g. be viewed on the gui logging page

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@265 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2005-06-10 09:19:24 +00:00
theli
c1a4e0dc28 *) changing reference to logger 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@252 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2005-06-09 10:44:55 +00:00
theli
d0083f845f *) changing reference to logger 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@251 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2005-06-09 10:39:09 +00:00
theli
1b5ae054f8 *) changing reference to logger 
*) parser will not be returned into pool if the parser was deactivated
   via gui

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@250 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2005-06-09 10:38:00 +00:00
theli
68f30811fa *) changing reference to logger 
*) bugfix in function getCachePath

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@249 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2005-06-09 10:36:39 +00:00
theli
fbbea813c5 *) changing references to logger 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@248 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2005-06-09 10:34:20 +00:00
orbiter
4574fa4ce7 bugfixes 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@224 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2005-06-08 15:28:29 +00:00
theli
83b41ef2f7 *) Adding timeouts for shutdown 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@223 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2005-06-08 13:44:25 +00:00
theli
ef6851798b *) changing thread priority while parsing a pdf file to avoid 100% CPU usage. 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@222 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2005-06-08 13:23:35 +00:00
orbiter
33f9315e58 implemented multithreading of indexing 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@221 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2005-06-08 13:19:05 +00:00
orbiter
ca3b4ccaf4 added snippet-routines (not yet finished) 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@218 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2005-06-08 00:52:24 +00:00
orbiter
ee0758fe4d bugfixes/empty-dir-deletion/snippet-test-activation 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@212 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2005-06-07 01:05:55 +00:00
orbiter
594c591223 changes towards 0.38 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@208 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2005-06-03 02:43:35 +00:00
orbiter
d8fdc2526e added experimental snipplet-generation (to be disabled for 0.38) 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@206 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2005-06-02 01:33:10 +00:00
orbiter
3771b10b89 implemented automated migration indexCache 0.37 -> indexAssortmentCluster 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@205 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2005-06-01 14:24:25 +00:00
orbiter
e89ded9e41 bugfixes 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@204 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2005-05-31 22:12:43 +00:00
orbiter
650ca3955a added flush-thread for index cache and added language-name mapping in Language_p 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@203 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2005-05-31 17:39:14 +00:00
orbiter
3d8a2ff937 enhanced parallelization of local/global/remote crawling 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@197 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2005-05-29 11:56:40 +00:00
orbiter
a05d738ea4 enhanced caching, removed bug causing outOfMemory 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@195 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2005-05-29 00:34:01 +00:00
orbiter
21110dcd5e fixed bugs with open files and caching 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@175 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2005-05-25 13:48:48 +00:00
orbiter
f8f8dd05db fixed "Too many open files" - bug 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@174 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2005-05-25 11:35:01 +00:00
theli
74eb21f62e *) adding image tag into rss template 
*) adding a xslt stylesheet so that the rss document can be viewed in a normal webbrowser
*) adding pubDate tag to each search item

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@173 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2005-05-25 08:47:34 +00:00
orbiter
5f90daa265 implemented localization environment 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@171 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2005-05-24 14:23:04 +00:00
theli
84f9d8f7f0 *) migrating ant build files to generate a single extension tar per default 
This is buildconfigureable by changing the extensionTarget Property in the build.properties file
*) Trying to solve "yacy.java template replacement / ant build failed" bug

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@169 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2005-05-24 09:02:04 +00:00
orbiter
fdd606c8c8 fixed bugs 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@168 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2005-05-23 23:50:35 +00:00
theli
8bd49ba535 *) setting root dir for all tar files properly 
*) automatically adding SVN Revision number to tar file name
*) introducing build.properties file that can be used to set the build version number and date

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@164 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2005-05-23 15:47:08 +00:00
orbiter
0c35171c85 assortment fine-tuning 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@163 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2005-05-23 15:07:16 +00:00
orbiter
76dc892017 refined assortment 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@159 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2005-05-23 13:15:28 +00:00
theli
0484c41a84 *) replacing system.xxx.println with logging statements 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@156 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2005-05-23 09:08:36 +00:00
theli
7994c485f1 *) Trying to set the document title properly 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@154 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2005-05-23 08:32:22 +00:00
theli
285936d778 *) trying to set document title properly 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@153 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2005-05-23 08:24:15 +00:00
theli
573a8e8047 *) setting document title properly 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@152 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2005-05-23 08:21:35 +00:00
orbiter
4b01ff7548 activated assortments, removed write-queues 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@151 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2005-05-22 23:59:52 +00:00
orbiter
e26ac60c3e modified assortment data structures 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@148 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2005-05-22 13:27:54 +00:00
orbiter
79be6f003d enhanced Assortment class 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@141 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2005-05-19 16:41:08 +00:00
theli
9ee3e69021 *) Solving "Warning: You did not close the PDF Document" problem when an OutOfMemory Exception occured ... 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@140 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2005-05-19 11:16:02 +00:00
orbiter
5c6147a54c introduced assortment structure (generalization of singletons) 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@139 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2005-05-18 21:52:17 +00:00
theli
73e297f30f *) adding proper default values for RealtimeParsableMimeTypes if something goes wrong with the configuration file 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@132 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2005-05-18 07:40:48 +00:00
theli
893a662329 *) Adding missing cast statement 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@127 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2005-05-17 08:40:27 +00:00
theli
361f05978d Multiple updates regarding the yacy seedUpload facility, 
optional content parsers, thread pool configuration ...

Please help me testing if everything works correct.

*) Migration of yacy seedUpload functionality
See: http://www.yacy-forum.de/viewtopic.php?t=256
- new uploaders can now be easily introduced because of a new modulare uploader system
- default uploaders are: none, file, ftp
- adding optional uploader for scp
- each uploader provides its own configuration file that will be 
  included into the settings page using the new template include feature
- Each uploader can define its libx dependencies. If not all needed libs are
  available, the uploader is deactivated automatically.

*) Migration of optional parsers
See: http://www.yacy-forum.de/viewtopic.php?t=198
- Parsers can now also define there libx dependencies
- adding parser for bzip compressed content
- adding parser for gzip compressed content
- adding parser for zip files
- adding parser for tar files
- adding parser to detect the mime-type of a file
  this is needed by the bzip/gzip Parser.java
- adding parser for rtf files
- removing extra configuration file yacy.parser
  the list of enabled parsers is now stored in the main config file

*) Adding configuration option in the performance dialog to configure
See: http://www.yacy-forum.de/viewtopic.php?t=267
- maxActive / maxIdle / minIdle values for httpd-session-threadpool
- maxActive / maxIdle / minIdle values for crawler-threadpool

*) Changing Crawling Filter behaviour
See: http://www.yacy-forum.de/viewtopic.php?p=2631

*) Replacing some hardcoded strings with the proper constants of the httpHeader class

*) Adding new libs to libx directory. This libs are
- needed by new content parsers
- needed by new optional seed uploader
- needed by SOAP API (which will be committed later)

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@126 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2005-05-17 08:25:04 +00:00
theli
ddc5675781 *) Correcting typo 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@120 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2005-05-14 11:14:34 +00:00
theli
d2c4e9a55e *) Implementing yacy forum wishlist item: "Pause Crawling" 
see: http://www.yacy-forum.de/viewtopic.php?t=48



git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@118 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2005-05-14 09:41:05 +00:00
orbiter
287d2e6f10 further enhanced caching (new cache flush methods) 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@111 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2005-05-12 23:33:48 +00:00
orbiter
376b917c91 fixed shut-down by stopYACY.sh 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@108 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2005-05-12 10:05:17 +00:00
orbiter
ea478f3975 enhanced indexing-caching 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@107 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2005-05-11 23:42:40 +00:00
orbiter
b4030e5023 implemented serverSwitchActions - action-hooks 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@105 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2005-05-11 14:58:03 +00:00
theli
6f4d2e5272 *) fixing replace bug. 
using 
      stringvar = stringvar.replace(xxx) 
   istead of 
      stringvar.replace()

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@101 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2005-05-10 12:17:50 +00:00
orbiter
10a4a2741d fixed missing close 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@94 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2005-05-08 20:39:11 +00:00
orbiter
db1da3345d introduced singleton-database 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@92 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2005-05-08 14:42:09 +00:00
orbiter
a9b22647dc fixed bug in indexDump.stack - generation 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@88 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2005-05-08 00:11:59 +00:00
orbiter
1d7fed87dc redesign of index caching - removed indexCache.db 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@86 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2005-05-07 21:11:18 +00:00
rramthun
3f85978519 Fixed one spelling mistake, limited input for ICQ numbers to 9 digits and made ICQ number in peer profiles clickable. 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@85 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2005-05-07 21:07:43 +00:00
theli
1dad015b0b *) Migration of Ant build files 
- now the ant build file has the same functionality as the makerelease build file
- from now on the ant build files can be used instead of the makerelease build script

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@84 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2005-05-06 08:43:11 +00:00
theli
2aa5fe8f50 *) Import statements reorganized 
Now it's easier to determine which class really uses which other class

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@82 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2005-05-05 05:32:19 +00:00
theli
351c86d5d9 *) Migration of optional Content Parser integration 
- each additional parser must be in a subpackage 
  of plasma.parser
- each parser must have its own ant build file (which will 
  be called automatically from the main build file)
- Calling the main build file results in building a separate 
  zip file for each optional parser. This zip file includes:
  + sources of the Parser.java
  + compiled classes of the Parser.java
  + needed additional libs (libx)
- To install an additional parser the user simply needs to
  extract the zip file listed above into his/her yacy directory.
- The configuration (enabling/disabling) of a parser can be done
  via the webinterface (currently the settings dialoge) and is
  done "on-the-fly". The installation can not be done "on-the-fly"
  at the moment because of classpath issues.
- The classpath of the linux startup/stop scripts is generated 
  automatically now (including all libraries from lib and libx).

*) Bugfix: File Extension was not calculated correctly by the crawler
   e.g.: file extension was accidentally: .php?param=value
   Corrected.

*) Adding additional parser for parsing of rss/atom feeds
- added needed libs to do this.

TODO:
- automatic building classpath for windows startup scripts


git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@78 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2005-05-03 09:47:56 +00:00
orbiter
d0010ff0b0 last changes for release 0.37 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@76 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2005-05-02 12:23:15 +00:00
orbiter
c7c6aaf06e many bug-fixes 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@73 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2005-04-30 01:22:46 +00:00
orbiter
48650c082c fixed 100%-CPU-Bug in plasmaCondenser 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@72 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2005-04-29 12:07:13 +00:00
orbiter
995673d795 several bugfixes 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@71 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2005-04-28 22:04:57 +00:00
orbiter
2de90020ed fixed caching+synchronization+brute-force-denial 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@67 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2005-04-27 21:09:40 +00:00
orbiter
9156fd53bc fixed bugs in last commit 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@65 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2005-04-26 15:47:33 +00:00
orbiter
e25f2354c2 removed synchronization and thread blockings 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@63 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2005-04-26 14:19:44 +00:00
theli
58a65b60bd *) synchronized keyword removed from function processLocalCrawling to avoid deadlocks. 
This synchronized keyword is not needed anymore because of the crawler jobqueue which
   is responsible for the synchronization now

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@60 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2005-04-26 06:59:36 +00:00
theli
65fc650109 *) plasmaCrawlLoader shutdown problem fixed (hopefully) 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@59 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2005-04-25 16:34:16 +00:00
orbiter
ba16da72b4 fixed not-working kelondroRecords-Cache 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@56 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2005-04-25 14:46:59 +00:00
orbiter
7fb645b0ab enhanced crawling performance, changed memory settings, new performace options 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@51 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2005-04-24 23:15:40 +00:00
theli
fd584c113c *) some minor changes 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@49 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2005-04-24 21:52:11 +00:00
theli
f44b219e44 *) Eclipse has accidentally copied in the wrong file header into the new files (because these headers were accidentally set as default for the whole workspace instead of the project) 
Fixed.

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@48 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2005-04-24 21:47:34 +00:00
theli
081ebd5517 *) I've accidentally used Java 5.0 syntax for enumerations 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@47 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2005-04-24 21:42:02 +00:00
theli
58b1a0ba40 *) adding an new package for extra content parsers 
*) adding content parser for
- pdf (using the pdf-box library)
- doc (using the textmining.org library)
*) adding a Interface for content parsers
*) adding a configuration file which can be used to configure which parser is used for which mimeType
*) Sempahore class was moved and renamed to serverSemaphore
*) Changing yacy shutdown behaviour
Buzy waiting loop for shutdown was removed and replaced with a blocking call (using the semaphore class mentioned above) to the new switchboard.waitForShutdown method.

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@46 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2005-04-24 21:24:53 +00:00
orbiter
8b31f9e202 enhanced shut-down behaviour & added experimental nio-wrapper for kelondroRA (not active yet) 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@44 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2005-04-23 13:00:56 +00:00
orbiter
00f223cfc1 fixed post-parsing (a case when the bluelist is empty) 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@41 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2005-04-21 17:13:43 +00:00
theli
c9c0a1f11c *) Trying to speedup local crawling 
- introduction of a threadpool for crawling
- introduction of a job queue to avoid buzy waiting for a free crawler slot

*) New classes added
- queue for receiving of crawler jobs
- semaphore class to do reader/writer synchronization (mutual exclusion)
- message object to hold all needed data about a crawler job

*) Trying to solve session-thread shutdown problem
- session thread stopped variable is now set from outside before interrupting the
  session thread.

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@39 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2005-04-21 10:31:40 +00:00
orbiter
97ec8d65e4 fixed makerelease & clean-up of dead code 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@33 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2005-04-19 14:04:16 +00:00
(no author)
1fec00bc24 *) Bugfix to avoid Nullpointer-Exceptions 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@30 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2005-04-19 10:39:58 +00:00
(no author)
f39812da91 *) Some performance improvements 
- many classes set to final
- implementation of a session-thread pool
- reusage of the server handler class (normally the httpd object)
  within the session thread
- implementation of a httpc object pool
- introduction of a linebuffer in httpd which can be reused
- reusing the properties table in the httpc
- added to apache libs (commons-collections, commons-pool) which 
  are needed for the object/thread pool implementation

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@26 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2005-04-19 06:55:57 +00:00
orbiter
b9203bdb50 bug fixes and code cleaning 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@22 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2005-04-15 14:18:14 +00:00
orbiter
c0807abd33 new crawl/proxy/cache design + fixes 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@18 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2005-04-13 23:00:20 +00:00
orbiter
e7d055b98e very experimental integration of the new generic parser and optional disabling of bluelist filtering in proxy. Does not yet work properly. To disable the disable-feature, the presence of a non-empty bluelist is necessary 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@17 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2005-04-13 15:52:00 +00:00
orbiter
96516fc9d8 fixed bugs (search+kelondroException, dns) 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@16 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2005-04-13 11:16:31 +00:00
orbiter
a87a17a3c8 prepared generic text parser environment 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@15 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2005-04-12 22:57:54 +00:00
orbiter
e374aca2cd enhanced exception handling in kelondro 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@14 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2005-04-12 15:45:50 +00:00
orbiter
89eb9a2292 fixed bug with crawl profiles 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@12 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2005-04-10 23:51:42 +00:00
orbiter
248077d3f0 initial load with yacy 0.36 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2005-04-07 19:19:42 +00:00