Commit Graph

3070 Commits

Author SHA1 Message Date
lotus
3fbfd5a78b * fix for non-changing offset on new search term
* dht-heap doesn't has to be deleted (5097), we simply write a new one on exit
* do not install YaCy in startup because a Windows-shutdown might corrupt something. Installing YaCy as a service would solve this.

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5099 6c8d7289-2bf4-0310-a012-ef5d649a1542
2008-09-02 15:09:31 +00:00
danielr
219b93df6a - fixed internal error after receiving chunked POST
- removed debug output
- added info for "501 Unknown" messages



git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5098 6c8d7289-2bf4-0310-a012-ef5d649a1542
2008-08-29 13:51:22 +00:00
lotus
c245c7a45e delete index.dhtin/out.heap if restore fails
see http://forum.yacy-websuche.de/viewtopic.php?p=9613#p9613

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5097 6c8d7289-2bf4-0310-a012-ef5d649a1542
2008-08-29 13:10:41 +00:00
danielr
cd19d0aee6 - added warnings for failed transferRWI (dht-in)
- fixed parseMultipart (uncompress gzipped body) (dht-in)
- fixed parseMultipart (using content-length only if uncompressed)
- better gzipped POST (chunked instead of content-length) (dht-out)



git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5096 6c8d7289-2bf4-0310-a012-ef5d649a1542
2008-08-29 09:42:39 +00:00
orbiter
df4ff423c4 added additional properties to query id's to distinguish search events better
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5093 6c8d7289-2bf4-0310-a012-ef5d649a1542
2008-08-28 21:15:59 +00:00
danielr
d6d9b0f14a fixed transferRWI.html 'Read timed out'
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5092 6c8d7289-2bf4-0310-a012-ef5d649a1542
2008-08-28 08:37:51 +00:00
danielr
e503158527 Proxy: fix for never ending loading after POST
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5091 6c8d7289-2bf4-0310-a012-ef5d649a1542
2008-08-27 20:46:34 +00:00
danielr
1a1d57e449 Proxy: added binary passthrough for POST
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5089 6c8d7289-2bf4-0310-a012-ef5d649a1542
2008-08-27 08:07:18 +00:00
apfelmaennchen
aa6ae77e5e - autoReCrawl: fix for filter settings
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5088 6c8d7289-2bf4-0310-a012-ef5d649a1542
2008-08-26 21:51:05 +00:00
apfelmaennchen
8ae29bad57 - fix to previous change of Crawl Profile Names
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5087 6c8d7289-2bf4-0310-a012-ef5d649a1542
2008-08-26 20:42:29 +00:00
apfelmaennchen
434104e4a0 - change Crawl profile name for autoreCrawl
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5085 6c8d7289-2bf4-0310-a012-ef5d649a1542
2008-08-26 18:08:48 +00:00
danielr
9ff4fc11da partial fix (images,audio,video) for proxy and content-type problem http://forum.yacy-websuche.de/viewtopic.php?f=5&t=1374
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5084 6c8d7289-2bf4-0310-a012-ef5d649a1542
2008-08-26 16:34:24 +00:00
lotus
0df2e47012 changed auto recrawl to comply with new date format
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5083 6c8d7289-2bf4-0310-a012-ef5d649a1542
2008-08-26 13:36:10 +00:00
lotus
d9d9c522a1 addendum to last commit
moved recrawl times for standard profiles to constants
calculate new specific dates in cleanup job

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5082 6c8d7289-2bf4-0310-a012-ef5d649a1542
2008-08-26 13:20:18 +00:00
lotus
480497f7c9 changed recrawl
use a specific date to define old documents
this solves an unwanted recrawl-loop during a running crawl

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5081 6c8d7289-2bf4-0310-a012-ef5d649a1542
2008-08-25 20:31:32 +00:00
orbiter
da1b0b2fc6 added two new classes that will be used for the new htcache
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5080 6c8d7289-2bf4-0310-a012-ef5d649a1542
2008-08-25 18:22:23 +00:00
orbiter
536e77e8b7 modifications towards a single database operation to read/write http header and cached file at once:
- removed distinction between header file types for http and ftp; ftp is simulated by using http properties
- removed all old resourceInfo classes that handled this distinction
- introduced a new distinction between http request and http response objects
- unified new response objects with two other object types that had been introduced elsewhere
- changed all servlet call methods to use the new http request header object type
- divided static object keys for http header properties into request and response types
- refactoring here and there (a large number of type changes and many methods merged/moved)


git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5079 6c8d7289-2bf4-0310-a012-ef5d649a1542
2008-08-25 18:11:47 +00:00
borg-0300
08cdf6db8a fix for wrong "VegaYacyB" peers
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5077 6c8d7289-2bf4-0310-a012-ef5d649a1542
2008-08-24 11:30:00 +00:00
danielr
4d937f6b21 fix for http://forum.yacy-websuche.de/viewtopic.php?f=5&t=1396
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5073 6c8d7289-2bf4-0310-a012-ef5d649a1542
2008-08-22 23:46:32 +00:00
apfelmaennchen
bd931a82f7 - added dynamic filters to autoReCrawl.conf
- Restrict to sub-path: sub
- Restrict to start-domain: dom

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5070 6c8d7289-2bf4-0310-a012-ef5d649a1542
2008-08-22 18:05:05 +00:00
apfelmaennchen
b3fc5e96a3 - removed unused import from bookmarksDB
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5067 6c8d7289-2bf4-0310-a012-ef5d649a1542
2008-08-20 21:26:06 +00:00
apfelmaennchen
bc048db7b6 - bugfix for bookmarksDB's rebuildDates()
- dates are now saved as String.valueOf(TimeStamp)
- it might be a good idea to delete (backup) bookmarkDates.db and restart YaCy to rebuild it 

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5066 6c8d7289-2bf4-0310-a012-ef5d649a1542
2008-08-20 21:25:05 +00:00
danielr
3c68905540 remove redundant null checks
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5065 6c8d7289-2bf4-0310-a012-ef5d649a1542
2008-08-20 08:37:39 +00:00
danielr
753a1ae430 - changed default browser from netscape to firefox
- fixed "Inefficient use of keySet iterator instead of entrySet iterator" [WMI_WRONG_MAP_ITERATOR, FindBugs]
- fixed some possible null pointer accesses


git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5063 6c8d7289-2bf4-0310-a012-ef5d649a1542
2008-08-20 07:54:56 +00:00
orbiter
7989335ed6 Preparations to replace the HTCache with a new storage data structure:
- refactoring of the HTCache (separation of cache entry)
- added new storage class for BLOBs. (not used yet, this is half-way to a new structure)

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5062 6c8d7289-2bf4-0310-a012-ef5d649a1542
2008-08-19 14:10:40 +00:00
danielr
be28af50f5 - fixed "yacy2yacy no proxy"-problem
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5058 6c8d7289-2bf4-0310-a012-ef5d649a1542
2008-08-17 10:16:32 +00:00
f1ori
f99c307eff * correct debian build dependencies
* add huge mem page detection in general initscript
* disable logging completely in jmimemagic-library


git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5056 6c8d7289-2bf4-0310-a012-ef5d649a1542
2008-08-14 21:01:21 +00:00
orbiter
bdae051d9a - extended new performance graph (better timing)
- added paths for new libraries in classpath for eclipse
- refactoring to remove compiler warnings (static access to finals variables)
- removed some unused import

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5055 6c8d7289-2bf4-0310-a012-ef5d649a1542
2008-08-13 10:37:53 +00:00
danielr
d9cea5ff23 removed annotations which broke the build with java 1.5
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5054 6c8d7289-2bf4-0310-a012-ef5d649a1542
2008-08-13 09:07:23 +00:00
danielr
a087090bbb fixed starting crawl results in "No parser available to parse mimetype 'application/octet-stream'"
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5047 6c8d7289-2bf4-0310-a012-ef5d649a1542
2008-08-10 11:31:40 +00:00
danielr
7e7e6a099a undo 5044
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5046 6c8d7289-2bf4-0310-a012-ef5d649a1542
2008-08-10 10:54:13 +00:00
danielr
f2d0bd7790 fix for NPE in JakartaHttpClient.setProxy
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5045 6c8d7289-2bf4-0310-a012-ef5d649a1542
2008-08-10 09:37:32 +00:00
danielr
bb6a6fc233 fixed 'FileUploadException Stream ended unexpectedly'
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5044 6c8d7289-2bf4-0310-a012-ef5d649a1542
2008-08-09 22:44:17 +00:00
danielr
8422ee5ec4 - fixed UnsupportedEncoding (in proxy) using defaultCharset if no characterEncoding can be determined
- serverFileUtils.copy* use now Charset instead of String
- added some warnings for ignored exceptions


git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5043 6c8d7289-2bf4-0310-a012-ef5d649a1542
2008-08-09 12:00:31 +00:00
hermens
3ac1988059 Add some sanity checks for invalid seeds
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5042 6c8d7289-2bf4-0310-a012-ef5d649a1542
2008-08-08 13:56:29 +00:00
hermens
cff4393f0c Fix HTCache so oldest Files get deleted first
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5041 6c8d7289-2bf4-0310-a012-ef5d649a1542
2008-08-08 08:06:06 +00:00
danielr
31d97f2b9f replaced httpd.parseMultipart() by a 'right' implementation
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5040 6c8d7289-2bf4-0310-a012-ef5d649a1542
2008-08-08 01:40:28 +00:00
danielr
621b473b18 * removed some warnings of findbugs (http://findbugs.sf.net)
- removed unnecessary code (unused variables, String.toString)
- corrected some calculations (cast int to double or long ;)
- improved little performance (using Integer.valueOf() instead of new Integer)
- log if some File-actions fail (mkdir(), delete(), ...) and some ignored exceptions
- finalized some (more) fields
- finally close some streams
- made inner classes static if not using environment
- generalized some equals (from specificClass to Object)
- fixed some potential nullpointer accesses


git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5039 6c8d7289-2bf4-0310-a012-ef5d649a1542
2008-08-06 19:43:12 +00:00
apfelmaennchen
0500b1179e added a 2 min start up delay to serverBusyThread autoReCrawl to avoid a Null Pointer Exception...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5035 6c8d7289-2bf4-0310-a012-ef5d649a1542
2008-08-05 05:50:15 +00:00
apfelmaennchen
e1574fe02e - added autoReCrawl folders to bookmarks (DATA/SETTINGS/autoReCrawl.conf)
- the serverBusyThread checks folders every 60 min. (==> autoReCrawl_idlesleep in yacy.conf)
- added option to create bookmarks from CrawlStart URL

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5033 6c8d7289-2bf4-0310-a012-ef5d649a1542
2008-08-04 20:43:36 +00:00
orbiter
ebb40d324b enhanced memory chart: shows now also the size of the word cache as third vector.
The PPM is now shown without a scale, but with a new anotation at the chart entry.

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5032 6c8d7289-2bf4-0310-a012-ef5d649a1542
2008-08-04 10:47:26 +00:00
danielr
17b7845eb5 * refactoring
- moved constants from plasmaSwitchboard to own class (all 232 ;)
- moved remoteProxy-Methods to httpRemoteProxyConfig, better names
- removed some unnecessary code (else-statements)
* formatting (correct indentation)
* minor bugfixes (due to findbugs.sf.net)
* hopefully fixed "missing quote" (announcing StringParts as UTF-8)


git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5031 6c8d7289-2bf4-0310-a012-ef5d649a1542
2008-08-02 13:57:00 +00:00
danielr
3bb870bfcd added final where possible
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5030 6c8d7289-2bf4-0310-a012-ef5d649a1542
2008-08-02 12:12:04 +00:00
lotus
7e92484400 fix for open browser on windows 2000
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5029 6c8d7289-2bf4-0310-a012-ef5d649a1542
2008-08-01 12:58:36 +00:00
f1ori
b0724e5ec0 * add config option to disable cookie monitoring (disabled by default)
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5028 6c8d7289-2bf4-0310-a012-ef5d649a1542
2008-07-30 21:19:06 +00:00
lotus
0b2f67577e Index Transfer:
- fix for chunk size calculation
- fix: if chunk size was 1, an infinite selection loop ran because no entries were found. if chunk size fails <=3 it will be set back to 500

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5023 6c8d7289-2bf4-0310-a012-ef5d649a1542
2008-07-27 18:53:51 +00:00
lotus
694084c570 fix for NPE on shutdown
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5021 6c8d7289-2bf4-0310-a012-ef5d649a1542
2008-07-27 06:59:56 +00:00
lotus
5f77f55ed7 possible fix for negative speed values
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5019 6c8d7289-2bf4-0310-a012-ef5d649a1542
2008-07-27 06:58:35 +00:00
orbiter
50ef5c406f - refactoring of robots parser (removed opaque Objects[] result vector)
- added Allow-component to robots result object

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5016 6c8d7289-2bf4-0310-a012-ef5d649a1542
2008-07-24 11:54:37 +00:00
danielr
7913bdb75b Flextable: filename in errormessage if inconsistent
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5015 6c8d7289-2bf4-0310-a012-ef5d649a1542
2008-07-24 08:22:36 +00:00