- harmonizing proxy exception handling
- adding malformed URL + blacklist check for http head method
- adding malformed URL check to http post method
- chunked encoding is now not used anymore for http post if clients
are http/0.9 or http/1.0 clients (same behaviour as already implemented for get)
- now an exception will be thrown on internal httpc errors to force an error output
to the client or a connection close. This should help to fix the "binary data in browser window" bug
*) plasmaSwitchboard.java
- fixing the following Bug
E 2005/09/03 18:02:42 PLASMA Could not index URL http://mis04.de/FAIL/snot.php: null
java.lang.NullPointerException
at de.anomic.plasma.plasmaSwitchboard.processResourceStack(plasmaSwitchboard.java:1000)
at de.anomic.plasma.plasmaSwitchboard.deQueue(plasmaSwitchboard.java:625)
at sun.reflect.GeneratedMethodAccessor2.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:585)
at de.anomic.server.serverInstantThread.job(serverInstantThread.java:95)
at de.anomic.server.serverAbstractThread.run(serverAbstractThread.java:243)
This bug could occure if the cached responseHeader is null
- getting the mimeType now from the parsed document instead of the responseHeader because the
mimeType could have been changed during content parsing (e.g. because of the mimetypeParser)
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@656 6c8d7289-2bf4-0310-a012-ef5d649a1542
See: http://www.yacy-forum.de/viewtopic.php?t=1118&highlight=xforwardedfor
*) httpc.java: Bugfix for incorrect http response statuscode parsing
In some situations the statustext whas chopped
*) Adding a lot of fileheaders containing YaCy copyright and license
*) httpd.java: Adding additional debugging http header that should help du detect
the "binary data in browser window" bug.
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@653 6c8d7289-2bf4-0310-a012-ef5d649a1542
Crawler StartURLs will now also added to the errorURL-DB if an error occures on this url
*) kelondroStack.java, plasmaSwitchboardQueue.java
Adding method which returns a list of all entries in the queue. This list is used by IndexCreate_p.java
instead of an iterator to display the indexing-list.
Advantages: avoid concurrent modifications of the list while displaying it.
Speedup because now we have to access only one sync function instead of multiple ones
(one for each entry)
*) IndexCreateIndexingQueue_p.java
Using new list() function of plasmaSwitchboardQueue
*) httpdFileHandler.java
If a servelet returns the special value "LOCATION" the httpFileHandler does a Redirection of
the Browser to the URL specified by the servelet. This can e.g. be used when a http get request is
used insead of a post request, but a refresh should not be allowed.
*) IndexCreateWWWLocalQueue_p.html
Now it's possible to delete single entries of the local crawler queue
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@626 6c8d7289-2bf4-0310-a012-ef5d649a1542
A typical useragent string now e.g. looks like:
Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.7.10; YaCy 0.401/00602; yacy.net) Gecko/20050716 Firefox/1.0.6
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@607 6c8d7289-2bf4-0310-a012-ef5d649a1542
looks very disordered? Inner classes and methods mixed together. Maybe the code
should be cleaned up a little bit?
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@503 6c8d7289-2bf4-0310-a012-ef5d649a1542
yacy-javadoc-documentation in doc/api. Just do ant create-doc and point your
favourite browser to doc/api/index.html. As most of the classes are not
documented right now this just gives a great overview of all classes.
Hopefully this helps stimulating the creation of
javadoc-insource-documentation.
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@502 6c8d7289-2bf4-0310-a012-ef5d649a1542
- Now it's possible to interrupt pending httpc-actions on server shutdown
- this is possible because of a newly introduced registration mechanism for
open sockets
*) yacyCore.java
- blocking peerPing threads can now be interrupted on server shutdown
*) serverCore.java
- restructuring shutdown code
*) error.html
- port number is now set correctly if port forwarding was enabled
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@389 6c8d7289-2bf4-0310-a012-ef5d649a1542
- Adding function to manually force peer ping to remote yacy peer
See:Network.html?page=4
- for debugging purpose only!
*) serverAbstractThread.java:
- Adding posibility to notify a server thread via a synchronization object
- this is needed e.g. by the port forwarding feature to send a notification
to the peerPing thread to redo peer-ping with the new ip/port Settings_p.html
*) Port Forwarding Feature (it should work now)
- adding a serverThread which is responsible to detect broken port forwarding
connections and to do reconnect if needed
- serverCore.java: moving port forwarding initialization into a separate function
- adding positility to configure the ssh port
- moving configuration section on the gui into a separate fieldset
- hello.java: only trying to do a second connect to the clientIp address during
peer handshake if either remote port forwarding is not enabled locally or
the clientIP is not equal to any local ip
*) httpdFileHandler.java:
- printout a more verbose errormessage
*) httpc.java
- allowing to deactivate content encoding from outside
*) plasmaCrawlWorker.java
- the crawler worker now tries to refetch the content of a website without
gzip content encoding if a gzip error occured
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@368 6c8d7289-2bf4-0310-a012-ef5d649a1542
via a command-line email program (e.g. sendmail) to a configured email address
- the configuration dialog is reachable via Settings_p.html#messageForwarding
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@332 6c8d7289-2bf4-0310-a012-ef5d649a1542
*) hello.java: reportedip my be empty at peer startup
*) httpc.java: adding method to determine if the connection was already closed or is broken
*) httpdProxyHandler.java: trying to do a better errorhandling
*) server/serverCore.java
- setting myseed ip-address and port correctly if port-forwarding is on
- doing a more failsafe close and adding some debugging output
*) yacyClient.java: adding some logging statements to allow a better detection of
"degraded to senior"-bug
*) yacyCore.java: restructuring publishMySeed
(@Orbiter: pleas take a look)
- to avoid buzy waiting
- to allow a gracefull shutdown on server shutdown
- new seed count was not calculated correctly in the previous version
*) yacySeedDB.java: host ip and port was not initialized correctly if port-forwarding
was activated
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@318 6c8d7289-2bf4-0310-a012-ef5d649a1542
*) bugfix in plasma/plasmaParser.java:
- parsers with missing dependencies wehre not ignored correctly
*) passing a logger instance to the parsers modules which can be used
for logging purposes by the parsers (not done yet)
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@276 6c8d7289-2bf4-0310-a012-ef5d649a1542
*) serverLog.java logging functions now also accept exceptions als
additional parameters.
The Stacktrace of this ecceptions will then be appended to the
logging message and can e.g. be viewed on the gui logging page
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@265 6c8d7289-2bf4-0310-a012-ef5d649a1542
access.log) file
*) splitting doGet function in separate functions for fulfilling
requests from cache and from web to make error handling easier
*) using connection property and httpHeader constants instead of
hardcoded strings whenever possible
*) sending back a proxy error message as body of every respond
containing a http error code
*) correcting problems of messages received from other proxies
containing 204, 304 status codes.
*) using chunked transfer encoding if the server has not set the
content length (e.g. because of gzip content encoding) but
the client has established a persistent connection to yacy.
This is only possible for http/1.1 clients. For http/1.0 clients
the connection will simply be closed on the end of the message.
*) removing unneeded functions (e.g. respondError) because of newly
introduced functions of httpd.java
*) removing hop by hop headers (according to rfc)
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@245 6c8d7289-2bf4-0310-a012-ef5d649a1542
which should help to detect transfer errors on yacy to yacy
communication
- not finished yet
*) removing unneeded functions (e.g. respondHeader) because newly
introduced functions in class httpd.java
*) httpdFileHandler.java now always sends back a proxy error message
as body of a response with an error code
*) adding support of gzip content encoding
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@244 6c8d7289-2bf4-0310-a012-ef5d649a1542
*) adding hashset for file-extensions that should not be transfered
using gzip content encoding
*) doing bugfixes on old keep-alive implementation
*) doing some additional http header validation according to rfc
*) doing all persistent connection detection in separate function now
*) doing server authentication in separate function now
*) doing proxy authentication in separate function now
*) simplifying GET, POST, HEAD functions because of new introduced
functions listed above
*) adding new function to handle empty request lines (which could
occure after post requests send via a persistent connection;
this depends on the used browser)
*) adding new function to handle unknown request methods by sending
a correct error message back to the client
*) setting correct content-length when sending back error messages
to the client
*) adding new functions that must be used by all http-Handler classes
to send
- a proxy error message
- a http header
back to the client
*) adding new function: shallTransportZipped
moved here from httpc
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@243 6c8d7289-2bf4-0310-a012-ef5d649a1542
- httpc
- response
*) simplifying gzip encoding
*) remembering http version of contacted server
(neede for later support of keep alive by httpc)
*) moving function shallTransportZipped to httpd.java
because this function is used multiple times
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@242 6c8d7289-2bf4-0310-a012-ef5d649a1542
*) Making Seed-Upload configuration more verbose.
*) Some Changes in SOAP Search API (not finished yet).
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@158 6c8d7289-2bf4-0310-a012-ef5d649a1542
optional content parsers, thread pool configuration ...
Please help me testing if everything works correct.
*) Migration of yacy seedUpload functionality
See: http://www.yacy-forum.de/viewtopic.php?t=256
- new uploaders can now be easily introduced because of a new modulare uploader system
- default uploaders are: none, file, ftp
- adding optional uploader for scp
- each uploader provides its own configuration file that will be
included into the settings page using the new template include feature
- Each uploader can define its libx dependencies. If not all needed libs are
available, the uploader is deactivated automatically.
*) Migration of optional parsers
See: http://www.yacy-forum.de/viewtopic.php?t=198
- Parsers can now also define there libx dependencies
- adding parser for bzip compressed content
- adding parser for gzip compressed content
- adding parser for zip files
- adding parser for tar files
- adding parser to detect the mime-type of a file
this is needed by the bzip/gzip Parser.java
- adding parser for rtf files
- removing extra configuration file yacy.parser
the list of enabled parsers is now stored in the main config file
*) Adding configuration option in the performance dialog to configure
See: http://www.yacy-forum.de/viewtopic.php?t=267
- maxActive / maxIdle / minIdle values for httpd-session-threadpool
- maxActive / maxIdle / minIdle values for crawler-threadpool
*) Changing Crawling Filter behaviour
See: http://www.yacy-forum.de/viewtopic.php?p=2631
*) Replacing some hardcoded strings with the proper constants of the httpHeader class
*) Adding new libs to libx directory. This libs are
- needed by new content parsers
- needed by new optional seed uploader
- needed by SOAP API (which will be committed later)
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@126 6c8d7289-2bf4-0310-a012-ef5d649a1542
so that a thread dump is more verbose
*) Moving code for transparent proxy support to a separate function
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@98 6c8d7289-2bf4-0310-a012-ef5d649a1542
- introduction of a threadpool for crawling
- introduction of a job queue to avoid buzy waiting for a free crawler slot
*) New classes added
- queue for receiving of crawler jobs
- semaphore class to do reader/writer synchronization (mutual exclusion)
- message object to hold all needed data about a crawler job
*) Trying to solve session-thread shutdown problem
- session thread stopped variable is now set from outside before interrupting the
session thread.
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@39 6c8d7289-2bf4-0310-a012-ef5d649a1542
can be used instead of a ByteArrayOutputStream
*) Using a serverByteBuffer for lineBuffering in class httpc
instead of a ByteArrayOutputStream
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@35 6c8d7289-2bf4-0310-a012-ef5d649a1542
- httpc: wrong error-message on 404
- httpc: error message was accidentally shown when object
was released from pool
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@31 6c8d7289-2bf4-0310-a012-ef5d649a1542
- many classes set to final
- implementation of a session-thread pool
- reusage of the server handler class (normally the httpd object)
within the session thread
- implementation of a httpc object pool
- introduction of a linebuffer in httpd which can be reused
- reusing the properties table in the httpc
- added to apache libs (commons-collections, commons-pool) which
are needed for the object/thread pool implementation
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@26 6c8d7289-2bf4-0310-a012-ef5d649a1542
