borg-0300
81cb8feb15
back to 649 :/
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@651 6c8d7289-2bf4-0310-a012-ef5d649a1542
2005-09-04 22:03:44 +00:00
borg-0300
5194511e8e
*) attempt to find bug
...
See: http://www.yacy-forum.de/viewtopic.php?t=1121
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@650 6c8d7289-2bf4-0310-a012-ef5d649a1542
2005-09-04 19:08:51 +00:00
theli
6991b9e2b9
*) Suppress stacktrace on crawler error for "Connection reset"
...
See: http://www.yacy-forum.de/viewtopic.php?p=9071
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@645 6c8d7289-2bf4-0310-a012-ef5d649a1542
2005-09-03 15:17:19 +00:00
theli
a47f9238fe
*) Blacklist is now also used by the crawler
...
See: http://www.yacy-forum.de/viewtopic.php?t=1069
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@642 6c8d7289-2bf4-0310-a012-ef5d649a1542
2005-09-02 12:09:45 +00:00
borg-0300
cc493ef8c1
Added change from Hermes
...
See: http://www.yacy-forum.de/viewtopic.php?t=1050
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@629 6c8d7289-2bf4-0310-a012-ef5d649a1542
2005-09-01 11:18:41 +00:00
theli
59b8a98c7e
*) Bugfix for suppressing of stacktrace in log on crawler error "MalformedURLException"
...
See: http://www.yacy-forum.de/viewtopic.php?p=8840
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@623 6c8d7289-2bf4-0310-a012-ef5d649a1542
2005-09-01 06:31:30 +00:00
theli
4fd5b95b1f
*) Renaming Logger function names to reflect the proper Java Logging API Loglevels
...
- please use logFine instead of logDebug
- please use logSevere instead of logFailure and logError
See: http://www.yacy-forum.de/viewtopic.php?p=8726#8726
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@615 6c8d7289-2bf4-0310-a012-ef5d649a1542
2005-08-30 21:32:59 +00:00
theli
6adf8a4bde
*) Renaming Logger function names to reflect the proper Java Logging API Loglevels
...
- please use logFine instead of logDebug
- please use logFailure instead of logError
See: http://www.yacy-forum.de/viewtopic.php?p=8726#8726
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@614 6c8d7289-2bf4-0310-a012-ef5d649a1542
2005-08-30 21:10:39 +00:00
theli
f19c09b227
*) Suppress stacktrace on crawler error for "MalformedURLException"
...
See: http://www.yacy-forum.de/viewtopic.php?p=8733#8733
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@613 6c8d7289-2bf4-0310-a012-ef5d649a1542
2005-08-30 20:25:07 +00:00
theli
9b818b1ce3
*) Pausing Crawlers if there is not enough space on disk
...
See: http://www.yacy-forum.de/viewtopic.php?p=8648
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@603 6c8d7289-2bf4-0310-a012-ef5d649a1542
2005-08-30 09:43:27 +00:00
theli
34790acf02
*) Bugfix for suppressing of stacktrace in log on crawler error "unknown host"
...
See: http://www.yacy-forum.de/viewtopic.php?p=8615#8615
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@600 6c8d7289-2bf4-0310-a012-ef5d649a1542
2005-08-30 06:24:23 +00:00
theli
af7b8f75bd
*) Making proxyAccessLogging configureable via yacy.logging file
...
- logging can be disabled now
- logging directory / filelimit / rotation count can be configured now
See: http://www.yacy-forum.de/viewtopic.php?t=965&postdays=0&postorder=asc&start=30#8280
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@595 6c8d7289-2bf4-0310-a012-ef5d649a1542
2005-08-29 11:31:58 +00:00
theli
cb1f11c96b
*) Suppress stacktrace on crawler error for "Unknown Host"
...
See: http://www.yacy-forum.de/viewtopic.php?p=8431
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@590 6c8d7289-2bf4-0310-a012-ef5d649a1542
2005-08-28 05:08:26 +00:00
theli
e338a13de3
*) Suppress stacktrace on crawler error for "Read timed out"
...
See: http://www.yacy-forum.de/viewtopic.php?p=8433
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@589 6c8d7289-2bf4-0310-a012-ef5d649a1542
2005-08-27 18:48:02 +00:00
theli
2e43e744de
*) Suppress stacktrace on crawler error for "connect timed out"
...
See: http://www.yacy-forum.de/viewtopic.php?p=8420
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@588 6c8d7289-2bf4-0310-a012-ef5d649a1542
2005-08-27 04:53:25 +00:00
theli
36cbe04e3e
*) Bugfix for Crawler Redirection Bug
...
See: http://www.yacy-forum.de/viewtopic.php?p=8422
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@587 6c8d7289-2bf4-0310-a012-ef5d649a1542
2005-08-27 04:36:13 +00:00
theli
17be77a468
*) Bugfix for "Crawler data will not be removed from htcache if content parsing failed"
...
See: http://www.yacy-forum.de/viewtopic.php?t=965&highlight=ramdisk
*) Making ACCEPT_LANGUAGE configureable for crawler
See: http://www.yacy-forum.de/viewtopic.php?p=8327
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@583 6c8d7289-2bf4-0310-a012-ef5d649a1542
2005-08-24 07:47:42 +00:00
theli
330eae7cf3
*) Normalizing CrawlerStartURL now before crawling is started
...
*) CrawlWorker also does a URL normalization now before following the redirection URL
*) CrawlWorker removes redirection URL correctly from noticeURL stack now
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@571 6c8d7289-2bf4-0310-a012-ef5d649a1542
2005-08-21 22:52:46 +00:00
theli
ea9a992f05
*) Before the crawler retries to download a URL it checks if the server is already doing a shutdown
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@554 6c8d7289-2bf4-0310-a012-ef5d649a1542
2005-08-17 11:36:48 +00:00
theli
ea26b84eed
*) Bugfix for http://www.yacy-forum.de/viewtopic.php?t=954
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@553 6c8d7289-2bf4-0310-a012-ef5d649a1542
2005-08-17 10:12:52 +00:00
orbiter
ba0a486328
moved printStackTrace() to logging
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@539 6c8d7289-2bf4-0310-a012-ef5d649a1542
2005-08-14 23:35:18 +00:00
theli
89c9faa89e
*) More graceful logging output in crawler
...
See: http://www.yacy-forum.de/viewtopic.php?t=894
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@511 6c8d7289-2bf4-0310-a012-ef5d649a1542
2005-08-10 06:15:47 +00:00
theli
b32e7c516c
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@507 6c8d7289-2bf4-0310-a012-ef5d649a1542
2005-08-09 09:07:19 +00:00
theli
86305f051d
*) Trying to solve "java.net.BindException: Address already in use: JVM_Bind" Problem
...
by retrying Socket bind
See: http://www.yacy-forum.de/viewtopic.php?p=6935
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@497 6c8d7289-2bf4-0310-a012-ef5d649a1542
2005-08-06 14:38:58 +00:00
theli
1d83d7e4d7
*) httpdFileHandler.java:
...
no stacktrace will be printed into log file for "Connection timed out" Errors now
See: http://www.yacy-forum.de/viewtopic.php?p=6381
*) plasmaCrawlWorker.java:
If a "Read timed out" error occurs while crawling a site, the failed crawl will be
retried.
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@493 6c8d7289-2bf4-0310-a012-ef5d649a1542
2005-08-04 11:05:04 +00:00
orbiter
91163db52e
fix for more time-related problems in proxy
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@486 6c8d7289-2bf4-0310-a012-ef5d649a1542
2005-08-03 00:52:32 +00:00
rramthun
eacff63eda
Typos...
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@482 6c8d7289-2bf4-0310-a012-ef5d649a1542
2005-08-02 16:09:19 +00:00
rramthun
6f2f54a312
Translation, spelling...
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@444 6c8d7289-2bf4-0310-a012-ef5d649a1542
2005-07-27 20:29:35 +00:00
rramthun
0f11399d16
Some corrections...
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@409 6c8d7289-2bf4-0310-a012-ef5d649a1542
2005-07-19 16:07:13 +00:00
orbiter
2f0d7ea8d3
removed htcache stati (superfluous now)
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@396 6c8d7289-2bf4-0310-a012-ef5d649a1542
2005-07-09 00:33:34 +00:00
orbiter
8b89c46afe
fixed problem with cache write
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@394 6c8d7289-2bf4-0310-a012-ef5d649a1542
2005-07-08 16:06:35 +00:00
orbiter
455ae9f55f
fixed htcache-store problem and due-time for remote crawls
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@393 6c8d7289-2bf4-0310-a012-ef5d649a1542
2005-07-08 15:17:50 +00:00
theli
55d10b864c
*) further improvements in shutdown behaviour
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@392 6c8d7289-2bf4-0310-a012-ef5d649a1542
2005-07-07 22:03:52 +00:00
orbiter
419f8fb398
fixed bugs/missing code regarding new crawl stack
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@384 6c8d7289-2bf4-0310-a012-ef5d649a1542
2005-07-07 01:38:49 +00:00
orbiter
858cd94299
replaced indexing ram-queue by file-based stack-queue
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@381 6c8d7289-2bf4-0310-a012-ef5d649a1542
2005-07-06 14:48:41 +00:00
theli
57c30f1d78
*) bugfix for usage of httpc without gzip content encoding
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@369 6c8d7289-2bf4-0310-a012-ef5d649a1542
2005-07-04 11:25:25 +00:00
theli
0e2c33ee55
*) Network.html/Network.java:
...
- Adding function to manually force peer ping to remote yacy peer
See:Network.html?page=4
- for debugging purpose only!
*) serverAbstractThread.java:
- Adding posibility to notify a server thread via a synchronization object
- this is needed e.g. by the port forwarding feature to send a notification
to the peerPing thread to redo peer-ping with the new ip/port Settings_p.html
*) Port Forwarding Feature (it should work now)
- adding a serverThread which is responsible to detect broken port forwarding
connections and to do reconnect if needed
- serverCore.java: moving port forwarding initialization into a separate function
- adding positility to configure the ssh port
- moving configuration section on the gui into a separate fieldset
- hello.java: only trying to do a second connect to the clientIp address during
peer handshake if either remote port forwarding is not enabled locally or
the clientIP is not equal to any local ip
*) httpdFileHandler.java:
- printout a more verbose errormessage
*) httpc.java
- allowing to deactivate content encoding from outside
*) plasmaCrawlWorker.java
- the crawler worker now tries to refetch the content of a website without
gzip content encoding if a gzip error occured
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@368 6c8d7289-2bf4-0310-a012-ef5d649a1542
2005-07-04 11:09:48 +00:00
theli
eee6322aaf
*) Adding redirection support to plasmaCrawlWorker.java
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@328 6c8d7289-2bf4-0310-a012-ef5d649a1542
2005-06-28 08:07:41 +00:00
theli
cd279907c0
*) Adding redirection support to plasmaCrawlWorker.java
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@327 6c8d7289-2bf4-0310-a012-ef5d649a1542
2005-06-28 08:01:26 +00:00
theli
fafda068f9
*) allowing crawler to process resources with statuscode 203
...
- this is needed if yacy is behind a second proxy
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@316 6c8d7289-2bf4-0310-a012-ef5d649a1542
2005-06-23 10:00:31 +00:00
orbiter
1e7f062350
many bugfixes, memory leak fixes, performance enhancements; new kelondroHashtable; activated snippets
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@313 6c8d7289-2bf4-0310-a012-ef5d649a1542
2005-06-23 02:07:45 +00:00
theli
9e47ba5ad6
*) adding missing calls for function close() to avoid "too many open file" bug
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@282 6c8d7289-2bf4-0310-a012-ef5d649a1542
2005-06-16 08:34:52 +00:00
orbiter
a5b40923b6
added word migration to assortments (start with 'java -classpath classes yacy -migratewords')
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@278 6c8d7289-2bf4-0310-a012-ef5d649a1542
2005-06-15 01:22:07 +00:00
theli
ee9e110366
*) removing old logging configuration properties from yacy.init
...
*) serverLog.java logging functions now also accept exceptions als
additional parameters.
The Stacktrace of this ecceptions will then be appended to the
logging message and can e.g. be viewed on the gui logging page
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@265 6c8d7289-2bf4-0310-a012-ef5d649a1542
2005-06-10 09:19:24 +00:00
theli
c1a4e0dc28
*) changing reference to logger
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@252 6c8d7289-2bf4-0310-a012-ef5d649a1542
2005-06-09 10:44:55 +00:00
orbiter
d8fdc2526e
added experimental snipplet-generation (to be disabled for 0.38)
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@206 6c8d7289-2bf4-0310-a012-ef5d649a1542
2005-06-02 01:33:10 +00:00
theli
361f05978d
Multiple updates regarding the yacy seedUpload facility,
...
optional content parsers, thread pool configuration ...
Please help me testing if everything works correct.
*) Migration of yacy seedUpload functionality
See: http://www.yacy-forum.de/viewtopic.php?t=256
- new uploaders can now be easily introduced because of a new modulare uploader system
- default uploaders are: none, file, ftp
- adding optional uploader for scp
- each uploader provides its own configuration file that will be
included into the settings page using the new template include feature
- Each uploader can define its libx dependencies. If not all needed libs are
available, the uploader is deactivated automatically.
*) Migration of optional parsers
See: http://www.yacy-forum.de/viewtopic.php?t=198
- Parsers can now also define there libx dependencies
- adding parser for bzip compressed content
- adding parser for gzip compressed content
- adding parser for zip files
- adding parser for tar files
- adding parser to detect the mime-type of a file
this is needed by the bzip/gzip Parser.java
- adding parser for rtf files
- removing extra configuration file yacy.parser
the list of enabled parsers is now stored in the main config file
*) Adding configuration option in the performance dialog to configure
See: http://www.yacy-forum.de/viewtopic.php?t=267
- maxActive / maxIdle / minIdle values for httpd-session-threadpool
- maxActive / maxIdle / minIdle values for crawler-threadpool
*) Changing Crawling Filter behaviour
See: http://www.yacy-forum.de/viewtopic.php?p=2631
*) Replacing some hardcoded strings with the proper constants of the httpHeader class
*) Adding new libs to libx directory. This libs are
- needed by new content parsers
- needed by new optional seed uploader
- needed by SOAP API (which will be committed later)
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@126 6c8d7289-2bf4-0310-a012-ef5d649a1542
2005-05-17 08:25:04 +00:00
orbiter
c7c6aaf06e
many bug-fixes
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@73 6c8d7289-2bf4-0310-a012-ef5d649a1542
2005-04-30 01:22:46 +00:00
orbiter
9156fd53bc
fixed bugs in last commit
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@65 6c8d7289-2bf4-0310-a012-ef5d649a1542
2005-04-26 15:47:33 +00:00
orbiter
e25f2354c2
removed synchronization and thread blockings
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@63 6c8d7289-2bf4-0310-a012-ef5d649a1542
2005-04-26 14:19:44 +00:00