orbiter
3879a0ecd0
replaced java.net.URL usage by use of new class de.anomic.net.URL
...
This shall be seen as an experiment to exclude all cases where
there could be a DNS lookup during URL comparisment.
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2290 6c8d7289-2bf4-0310-a012-ef5d649a1542
2006-07-13 01:21:53 +00:00
orbiter
d2bb3f442e
fixed timing problem causing a division by zero exception
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2287 6c8d7289-2bf4-0310-a012-ef5d649a1542
2006-07-11 23:43:25 +00:00
allo
2bdf1fc360
totalPPM
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2282 6c8d7289-2bf4-0310-a012-ef5d649a1542
2006-07-08 22:41:48 +00:00
orbiter
671fd9a5c9
work towards new indexing database structure
...
(no effect on current functionality yet)
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2277 6c8d7289-2bf4-0310-a012-ef5d649a1542
2006-07-04 14:47:27 +00:00
orbiter
92f4cb4d73
added option to configure the start-up delay time for kelondro database files.
...
the start-up delay is used to pre-load the database node cache
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2276 6c8d7289-2bf4-0310-a012-ef5d649a1542
2006-07-03 23:57:33 +00:00
orbiter
85d575e928
enhancements to kelondroRow and kelondroColumn
...
these are changes towards a better indexURLEntry implementation
which are needed for the new database structures
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2268 6c8d7289-2bf4-0310-a012-ef5d649a1542
2006-07-02 01:26:06 +00:00
orbiter
66964dc015
removed high/med/low from kelondroRecords cache control.
...
this was done because testing showed that cache-delete operations
slowed down record access most, even more that actual IO operations.
Cache-delete operations appeared when entries were shifted from low-priority
positions to high-priority positions. During a fill of x entries to a database,
x/2 delete situation happen which caused two or more delete operations.
removing the cache control means that these delete operations are not
necessary any more, but it is more difficult to decide which cache elements
shall be removed in case that the cache is full. There is not yet a stable
solution for this case, but the advantage of a faster cache is more important
that the flush problem.
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2244 6c8d7289-2bf4-0310-a012-ef5d649a1542
2006-06-25 10:31:38 +00:00
hermens
d4645062bc
Correct usage of vhost in wget/wput requests:
...
- yacyClient: don't use own .yacyh domain in requests, instead use .yacyh domain of target peer for everything but ranking distribution
- natLib: use full hostname instead of just SLD.TLD
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2232 6c8d7289-2bf4-0310-a012-ef5d649a1542
2006-06-21 14:25:27 +00:00
theli
4ca0857c0c
*) Index transfer now considers the pause time send by busy peers during
...
index transfer / index distribution
See: http://www.yacy-forum.de/viewtopic.php?p=22647#22491
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2205 6c8d7289-2bf4-0310-a012-ef5d649a1542
2006-06-14 09:40:42 +00:00
orbiter
eda055e729
- enhanced kelondroCollection speed
...
- tested + debugged kelondroCollection
- inserted the new kelondroCollection Object as indexing class for kelondroFlexTable
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2183 6c8d7289-2bf4-0310-a012-ef5d649a1542
2006-06-07 15:40:29 +00:00
orbiter
c36e9fc8d3
full integration of kelondroRow
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2167 6c8d7289-2bf4-0310-a012-ef5d649a1542
2006-06-02 12:45:57 +00:00
orbiter
4a907a570f
1st step to migrate kelondroTree to usage of kelondroRow instead of byte[][]
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2162 6c8d7289-2bf4-0310-a012-ef5d649a1542
2006-05-31 23:31:46 +00:00
orbiter
3c3c047d0a
integrated kelondroRow into kelondroStack
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2156 6c8d7289-2bf4-0310-a012-ef5d649a1542
2006-05-30 15:28:05 +00:00
orbiter
5041d330ce
refactoring
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2150 6c8d7289-2bf4-0310-a012-ef5d649a1542
2006-05-28 11:44:50 +00:00
orbiter
7b3b12888c
refactoring: integrated indexContainer abstraction layer
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2149 6c8d7289-2bf4-0310-a012-ef5d649a1542
2006-05-28 01:09:31 +00:00
orbiter
4d8f8ba384
added cache-performance analysis for node caches
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2140 6c8d7289-2bf4-0310-a012-ef5d649a1542
2006-05-24 09:40:08 +00:00
orbiter
a930be4ba3
refactoring of index management:
...
generalized the index entry
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2121 6c8d7289-2bf4-0310-a012-ef5d649a1542
2006-05-19 23:19:20 +00:00
orbiter
82b2bc6932
patch for index-transfer DoS problem
...
see http://www.yacy-forum.de/viewtopic.php?p=21627#21627
note that this function will make the index-transfer functionality void
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2114 6c8d7289-2bf4-0310-a012-ef5d649a1542
2006-05-18 22:24:51 +00:00
orbiter
a474669338
start with refactoring of index management
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2110 6c8d7289-2bf4-0310-a012-ef5d649a1542
2006-05-16 16:11:55 +00:00
orbiter
19c6e67003
- added generation of news for blog entries
...
- added display of blog banner in Network view
- extended lifetime of blog, wiki and crawl news
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2102 6c8d7289-2bf4-0310-a012-ef5d649a1542
2006-05-15 20:48:39 +00:00
auron_x
55ea4cbfe6
*)reverted patch for memory-display issue
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2095 6c8d7289-2bf4-0310-a012-ef5d649a1542
2006-05-14 18:09:28 +00:00
auron_x
53d9ab6db7
*)fixed bug in PerformanceMemory_p.java which caused negative memory-values on big peers
...
see http://www.yacy-forum.de/viewtopic.php?t=2370
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2091 6c8d7289-2bf4-0310-a012-ef5d649a1542
2006-05-14 08:42:41 +00:00
orbiter
35995cf8c7
some enhancements to caching and kelondroRA-methods
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2081 6c8d7289-2bf4-0310-a012-ef5d649a1542
2006-05-10 21:36:44 +00:00
orbiter
015d044c25
tried to fix some problems with latest changes to httpc
...
very experimental!
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2078 6c8d7289-2bf4-0310-a012-ef5d649a1542
2006-05-10 16:01:14 +00:00
orbiter
55c5b41bd0
modified kelondroDyn to work better with new object caches
...
(removed own single object cache)
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2077 6c8d7289-2bf4-0310-a012-ef5d649a1542
2006-05-10 13:57:31 +00:00
orbiter
26e3216bcc
update to profile fetch behavior
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2076 6c8d7289-2bf4-0310-a012-ef5d649a1542
2006-05-10 09:37:18 +00:00
orbiter
3e31820c3d
- corrections to PerformanceMemory display of object cache
...
- configuration of object cache size in kelondroTree initializer
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2075 6c8d7289-2bf4-0310-a012-ef5d649a1542
2006-05-10 09:08:42 +00:00
orbiter
fd7c17e624
added virtual host support:
...
all yacy-to-yacy communication now send the <peer-hexhash>.yacyh
virtual domain inside the http 'Host' property field.
This shall enable running a yacy peer on a virtual host.
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2074 6c8d7289-2bf4-0310-a012-ef5d649a1542
2006-05-09 13:11:00 +00:00
orbiter
29b1b0823c
added monitoring of new object cache to performanceMemory page
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2072 6c8d7289-2bf4-0310-a012-ef5d649a1542
2006-05-09 10:03:12 +00:00
orbiter
fe4ad214f1
configuration of time-out for fetching seed-lists during bootstraping
...
(weil das beim linuxtag nur auf meinem Notebook nicht funtioniert)
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2054 6c8d7289-2bf4-0310-a012-ef5d649a1542
2006-05-03 21:27:30 +00:00
orbiter
60e5aff9fc
some enhancements to the remote crawl trigger
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2030 6c8d7289-2bf4-0310-a012-ef5d649a1542
2006-04-20 11:53:15 +00:00
orbiter
dbe96e6541
added hand-over of search filter and prefer ranking to yacy protocol
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2029 6c8d7289-2bf4-0310-a012-ef5d649a1542
2006-04-20 10:15:00 +00:00
orbiter
bd283b8443
fixed bugs:
...
- null pointer exception during startup of a robinson-configured peer
- wrong time calculation of default value of re-crawl option
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2005 6c8d7289-2bf4-0310-a012-ef5d649a1542
2006-04-06 16:28:28 +00:00
orbiter
a469874e3f
added and fixed time-out behaviour during search
...
see also: http://www.yacy-forum.de/viewtopic.php?p=19823#19823
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1986 6c8d7289-2bf4-0310-a012-ef5d649a1542
2006-04-02 20:40:07 +00:00
rramthun
42b0b10a95
-Adding Windows Media to types which are not sended compressed
...
-Renaming writeandzip to writeandgzip to avoid confusion about type of compression
-Adding new startup message to windows script
-The usual language "enhancements" ;-)
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1953 6c8d7289-2bf4-0310-a012-ef5d649a1542
2006-03-23 20:12:23 +00:00
orbiter
63f39ac7b5
added 3 new crawling steering options:
...
- re-crawl by age of page (enter in minutes)
- auto-domain-filter
- maximum number of pages per domain
NOT YET TESTED!
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1949 6c8d7289-2bf4-0310-a012-ef5d649a1542
2006-03-23 16:05:16 +00:00
orbiter
1f4412a146
adopted isListed to discussed new behavior as discussed (url, getFile)
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1940 6c8d7289-2bf4-0310-a012-ef5d649a1542
2006-03-20 22:31:59 +00:00
orbiter
59fc55ea1e
added checks to protect peers from wrong seeds
...
see also: http://www.yacy-forum.de/viewtopic.php?p=19249#19249
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1939 6c8d7289-2bf4-0310-a012-ef5d649a1542
2006-03-20 20:31:51 +00:00
borg-0300
d808765087
something Javadoc
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1929 6c8d7289-2bf4-0310-a012-ef5d649a1542
2006-03-18 21:47:04 +00:00
orbiter
488a0ed580
replaced old keyIterator and rowIterator by buffered iterators
...
that are synchronized with database access
Main change is done in kelondroTree, other classes are only adoptions
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1918 6c8d7289-2bf4-0310-a012-ef5d649a1542
2006-03-17 23:43:24 +00:00
orbiter
3286b1f498
re-organisation of lurl-creation and -stacking
...
this was necessary to prevent useless write to the database
in case of blacklist appearance of the url
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1905 6c8d7289-2bf4-0310-a012-ef5d649a1542
2006-03-17 10:16:07 +00:00
theli
5ee0125046
*) adding possibility to configure the server port for seed uploading via scp.
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1861 6c8d7289-2bf4-0310-a012-ef5d649a1542
2006-03-08 16:34:05 +00:00
allo
2065138a41
reverted last Commit.
...
@Orbiter: kannst du dich dann mal drum kümmern, wenn ich versuche die ganze Sache ans Laufen zu bringen, hast du jedesmal was dagegen. Dann mach du es bitte, du wirst ja wissen was du willst...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1860 6c8d7289-2bf4-0310-a012-ef5d649a1542
2006-03-08 13:35:38 +00:00
allo
7afa5c1b8e
staticIP fix
...
tried to solve http://www.yacy-forum.de/viewtopic.php?p=18663#18663
D 2006/03/08 07:08:20 YACY yacyClient.publishMySeed mySeed error - not proper: IP is not proper: -UNRESOLVED_PATTERN-
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1859 6c8d7289-2bf4-0310-a012-ef5d649a1542
2006-03-08 12:23:26 +00:00
orbiter
a4682e2810
fixed problems in basic config and added language setting
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1799 6c8d7289-2bf4-0310-a012-ef5d649a1542
2006-03-02 22:25:46 +00:00
orbiter
ce5274c194
yacybot user agent
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1786 6c8d7289-2bf4-0310-a012-ef5d649a1542
2006-02-28 19:08:58 +00:00
allo
3b7e66ab48
staticIP should now work
...
(with resolved Conflict)
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1785 6c8d7289-2bf4-0310-a012-ef5d649a1542
2006-02-28 12:31:57 +00:00
orbiter
f96238e9be
cosmetic
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1778 6c8d7289-2bf4-0310-a012-ef5d649a1542
2006-02-27 22:25:27 +00:00
orbiter
eaffcfefe2
* added more ranking attributes (without function; this will be added later)
...
* added ranking coefficient transmission to remote peer (without evaluation on server side, will be added later)
* changed ranking coefficients slightly
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1770 6c8d7289-2bf4-0310-a012-ef5d649a1542
2006-02-26 11:30:37 +00:00
theli
468ca5b0e6
*) Bugfix for url.toString problem in yacyClient crawlOrder
...
Thanks to Stephan for the advice
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1737 6c8d7289-2bf4-0310-a012-ef5d649a1542
2006-02-22 07:18:27 +00:00
borg-0300
2ee5796e76
added keywords, properties;
...
change '\n' -> '\r\n'
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1720 6c8d7289-2bf4-0310-a012-ef5d649a1542
2006-02-21 00:30:38 +00:00
borg-0300
149409ba5c
move description -> javadoc
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1716 6c8d7289-2bf4-0310-a012-ef5d649a1542
2006-02-20 23:15:39 +00:00
orbiter
2c4e4ae6a2
further refactoring of dht selection, transfer and flushing
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1707 6c8d7289-2bf4-0310-a012-ef5d649a1542
2006-02-19 23:47:45 +00:00
theli
651bce8e2f
*) adding missing function to transmit url chunks for crawl-order jobs
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1680 6c8d7289-2bf4-0310-a012-ef5d649a1542
2006-02-17 11:04:35 +00:00
theli
9b941fb773
*) bugfix for usage of yacy with extended port binding (e.g. #eth0:8080, 192.168.0.1:8080, etc.)
...
- port was reported incorrectly to other peers
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1678 6c8d7289-2bf4-0310-a012-ef5d649a1542
2006-02-17 10:53:20 +00:00
orbiter
218cd6561c
fixed problem with wrong hash length in file share
...
see: http://www.yacy-forum.de/viewtopic.php?p=16565#16565
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1658 6c8d7289-2bf4-0310-a012-ef5d649a1542
2006-02-15 22:12:53 +00:00
allo
9d6bae77ee
the best PPM ever! ;-p
...
Now stopped indexingstop is recognized as 0 PPM.
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1641 6c8d7289-2bf4-0310-a012-ef5d649a1542
2006-02-14 17:29:46 +00:00
hydrox
a627162f13
*)fixed logginglevel for Debugmsg
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1585 6c8d7289-2bf4-0310-a012-ef5d649a1542
2006-02-09 09:10:49 +00:00
hermens
5f5eee1ae9
*) replace System.out.println with log
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1540 6c8d7289-2bf4-0310-a012-ef5d649a1542
2006-02-05 01:17:08 +00:00
orbiter
eab1805bca
refactoring: plasmaSearchProfile -> plasmaSearchTimingProfile
...
This was made to distiguish this profile from the
(to-be-implemented) plasmaSeachOrderProfile
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1538 6c8d7289-2bf4-0310-a012-ef5d649a1542
2006-02-04 23:11:31 +00:00
hermens
bb1664b63e
*) Remove workaround from SVN 1472: It is not needed anymore
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1500 6c8d7289-2bf4-0310-a012-ef5d649a1542
2006-01-31 00:45:48 +00:00
orbiter
3419b3bcdd
fix for bug that caused the peer-counter problem.
...
See http://www.yacy-forum.de/viewtopic.php?p=16016#16016
The kelondroDyn now uses a generic fill character.
kelondroDyn-Tables containing peer/word/url-hashes must not use '_'
as fill character.
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1498 6c8d7289-2bf4-0310-a012-ef5d649a1542
2006-01-30 22:16:55 +00:00
allo
9bc62bfdac
typo
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1497 6c8d7289-2bf4-0310-a012-ef5d649a1542
2006-01-30 20:57:11 +00:00
allo
cdc595ab67
another ppm fix. no more unknown ppm
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1496 6c8d7289-2bf4-0310-a012-ef5d649a1542
2006-01-30 20:42:02 +00:00
orbiter
fa90c3ca7a
- removed some usage of indexEntity
...
- changed index collection process: indexes are not first flushed to indexEntity,
but now collected directly from ram cache and assortments
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1489 6c8d7289-2bf4-0310-a012-ef5d649a1542
2006-01-30 12:42:06 +00:00
orbiter
03c65742ba
changes towards the new index storage scheme:
...
- replaced usage of temporary IndexEntity by EntryContainer
- added more attributes to word index
- added exact-string search (using quotes in query)
- disabled writing into WORDS during search; EntryContainers are used instead
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1485 6c8d7289-2bf4-0310-a012-ef5d649a1542
2006-01-30 00:42:38 +00:00
allo
a6245a302f
even better ppm ;-)
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1481 6c8d7289-2bf4-0310-a012-ef5d649a1542
2006-01-28 19:22:30 +00:00
hermens
2d1283da34
This is an extremely ugly workaround for an incompatibility between yacySeed hashes and kelondroDyn keys
...
See: http://www.yacy-forum.de/viewtopic.php?p=15955#15955
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1472 6c8d7289-2bf4-0310-a012-ef5d649a1542
2006-01-28 15:26:56 +00:00
allo
7197f171d3
better ppm calculation
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1464 6c8d7289-2bf4-0310-a012-ef5d649a1542
2006-01-27 11:51:27 +00:00
orbiter
b946e28e61
some ranking enhancements
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1460 6c8d7289-2bf4-0310-a012-ef5d649a1542
2006-01-27 02:48:27 +00:00
hermens
ad0de69607
Yet another bug fix for svn 1441. It should work now.
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1443 6c8d7289-2bf4-0310-a012-ef5d649a1542
2006-01-25 13:32:04 +00:00
hermens
58fd40e1c1
Aaargh
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1442 6c8d7289-2bf4-0310-a012-ef5d649a1542
2006-01-25 13:28:03 +00:00
hermens
b08af0c2cb
*) Force download of seed file when checking upload success
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1441 6c8d7289-2bf4-0310-a012-ef5d649a1542
2006-01-25 13:25:48 +00:00
hermens
66c889138e
*) Bugfix: Principals are reported back as 'principal', so IWasAccessed should also be true
...
*) make it easier to include legacy peers switching between timezones +0100 and +0200
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1438 6c8d7289-2bf4-0310-a012-ef5d649a1542
2006-01-25 01:50:24 +00:00
hermens
a2e2d583f9
*) small bugfix regarding peerPingMaxRunning
...
*) beautify log
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1419 6c8d7289-2bf4-0310-a012-ef5d649a1542
2006-01-24 01:16:22 +00:00
rramthun
a4e90c4b11
Fixed spelling bug.
...
I think this is important for other programmers, who don't make the same mistake as the original author.
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1417 6c8d7289-2bf4-0310-a012-ef5d649a1542
2006-01-23 15:08:39 +00:00
hermens
62ab8d18c1
*) Bugfix for peer sorting method. This seems to cause funny side effects in the SeedDB
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1412 6c8d7289-2bf4-0310-a012-ef5d649a1542
2006-01-23 01:42:26 +00:00
hermens
75b268f16d
*) use majority voting for peer type decision
...
*) reduce the number of peer pings sent out
see: http://www.yacy-forum.de/viewtopic.php?t=1748
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1411 6c8d7289-2bf4-0310-a012-ef5d649a1542
2006-01-22 23:14:37 +00:00
orbiter
f14d49fae9
enhancements, bugfixes and additions to word index attribute storage
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1392 6c8d7289-2bf4-0310-a012-ef5d649a1542
2006-01-22 00:07:00 +00:00
orbiter
f4ffa9aee5
- implemented more attributes to index entries
...
- implemented hand-over of new word index attributes during remote search
- implemented word-distance computation during search
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1382 6c8d7289-2bf4-0310-a012-ef5d649a1542
2006-01-20 15:14:21 +00:00
orbiter
3ec7177f0f
added seed flag 'AcceptCitationReference'
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1374 6c8d7289-2bf4-0310-a012-ef5d649a1542
2006-01-18 14:52:24 +00:00
orbiter
9544c47684
added some UTF-8 handling.
...
hope this will help somehow.. for shure not THE solution to our UTF-8 problem
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1308 6c8d7289-2bf4-0310-a012-ef5d649a1542
2006-01-10 16:48:59 +00:00
orbiter
9086261476
refactoring of base64 encoding:
...
the kelondro database needs specific information about the order of
base64-encoded keys. Since no other package depends on base64
(only the httpd uses base64 for encryption, but does not need to encode these strings)
it is good to move base64 encoding to the new ordering classes in kelondro.
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1284 6c8d7289-2bf4-0310-a012-ef5d649a1542
2006-01-04 00:39:00 +00:00
orbiter
b3dca06bb1
added location column to network pages.
...
The location is computed from the userAgent string of connecting peers.
Therefore this information is not available right after start-up.
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1241 6c8d7289-2bf4-0310-a012-ef5d649a1542
2005-12-22 01:01:46 +00:00
orbiter
0c762daf4b
better startup failure handling
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1205 6c8d7289-2bf4-0310-a012-ef5d649a1542
2005-12-12 23:59:58 +00:00
orbiter
37f88b4017
code cleanup
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1176 6c8d7289-2bf4-0310-a012-ef5d649a1542
2005-12-06 23:51:29 +00:00
orbiter
ec2b39c1ce
code cleanup
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1175 6c8d7289-2bf4-0310-a012-ef5d649a1542
2005-12-06 22:30:15 +00:00
theli
44fa94ac52
*) Modifications for dbImport functionality
...
- dbImporter threads are now shutdown by the switchboard on server shutdown
- adding possibility to pause a importer thread via GUI
- Bugfix for abort function
See: http://www.yacy-forum.de/viewtopic.php?p=13363#13363
*) Modification of content parser configuration
- now it's possible to configure which parsers should be enabled for the proxy,
crawler, icap, etc. separately
-
*) htmlFilterContentScraper.java
- adding regular expression to normalize URLs containing /../ and /./ parts
*) httpc.java
- adding functionality to unzip gzipped content
- requested by roland: should be used later to allow gzipped seed lists
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1170 6c8d7289-2bf4-0310-a012-ef5d649a1542
2005-12-06 10:41:19 +00:00
orbiter
3d8a5ae652
code cleanup
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1166 6c8d7289-2bf4-0310-a012-ef5d649a1542
2005-12-05 14:24:13 +00:00
orbiter
a04930f025
code cleanup
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1158 6c8d7289-2bf4-0310-a012-ef5d649a1542
2005-12-04 23:51:28 +00:00
theli
b604654c25
*) Adding possibility to do a settings migration on yacy startup
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1149 6c8d7289-2bf4-0310-a012-ef5d649a1542
2005-12-01 13:39:38 +00:00
orbiter
24dc0e0760
implemented cr-file processing and further transmission steps
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1099 6c8d7289-2bf4-0310-a012-ef5d649a1542
2005-11-17 01:59:01 +00:00
allo
f8f9d509d5
removed dead Code
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1078 6c8d7289-2bf4-0310-a012-ef5d649a1542
2005-11-14 12:48:14 +00:00
orbiter
a3fd0069f5
fixed bug in kelondroTree node iteration (rotation did not work)
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1072 6c8d7289-2bf4-0310-a012-ef5d649a1542
2005-11-14 00:23:20 +00:00
allo
889de6686c
Migration in yacyVersion
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1070 6c8d7289-2bf4-0310-a012-ef5d649a1542
2005-11-13 15:12:48 +00:00
orbiter
6e81f2580d
try to fix bug with storage of settings
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1058 6c8d7289-2bf4-0310-a012-ef5d649a1542
2005-11-11 08:41:13 +00:00
orbiter
79818a320f
introduced citation-rank transmission protocol and activate transport for anonymisation
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1055 6c8d7289-2bf4-0310-a012-ef5d649a1542
2005-11-10 23:48:20 +00:00
theli
6f8d7d3bcd
*) Adding first version of YaCy bookmarklet
...
- this can be used to easily crawl a webpage which is currently opened in the browser
- to get the bookmarklet javascript simply call http://localhost:8000/QuickCrawlLink_p.html
and drag and drop the link shown to your Browsers Toolbar/Link-Bar.
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1053 6c8d7289-2bf4-0310-a012-ef5d649a1542
2005-11-08 12:14:51 +00:00
orbiter
d2731418bf
added creation of global ranking files and changed url normal form usage
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1046 6c8d7289-2bf4-0310-a012-ef5d649a1542
2005-11-07 12:33:02 +00:00
theli
b8ceb1ffde
*) Adding better https support for crawler
...
- solving problems with unkown certificates by implementing a dummy trust Manager
- adding https support to robots-parser
- Seed File can now be downloaded from https resources
- adapting plasmaHTCache.java to support https URLs properly
*) URL Normalization
- sub URLs are now normalized properly during indexing
- pointing urlNormalForm function of plasmaParser to htmlFilterContentScraper function
- normalizing URLs which were received by a crawlOrder request
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1024 6c8d7289-2bf4-0310-a012-ef5d649a1542
2005-11-03 15:28:37 +00:00
borg-0300
d2507c6081
rename setJunior()... to orJunior()...,
...
added javadoc,
added getPeerType(), setIP(), setPort(String port)
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1023 6c8d7289-2bf4-0310-a012-ef5d649a1542
2005-11-03 14:16:16 +00:00
borg-0300
e3179a6394
added getOwnSeedFile()
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1022 6c8d7289-2bf4-0310-a012-ef5d649a1542
2005-11-03 14:07:58 +00:00
hydrox
cb69047b91
*)cleanup access static methods and fields
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1016 6c8d7289-2bf4-0310-a012-ef5d649a1542
2005-11-02 17:56:26 +00:00
hydrox
56b9f34411
*)removed unused imports
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1015 6c8d7289-2bf4-0310-a012-ef5d649a1542
2005-11-02 16:30:45 +00:00
hydrox
295aff52a3
*)added offline-browsing-support (onlineMode=0)
...
*)online-mode now can be changed in Status.html
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1010 6c8d7289-2bf4-0310-a012-ef5d649a1542
2005-10-31 12:25:40 +00:00
orbiter
b058ecf0bc
refactoring of image-generation; added experimental PNG encoder (not active now)
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1008 6c8d7289-2bf4-0310-a012-ef5d649a1542
2005-10-31 02:43:55 +00:00
theli
e58e85363d
*) Bugfix for ConcurrentModificationException while operating on seed properties
...
*) Bugfix for YACY database inconsistency (no more elements available in db '...seed.new.db'), re-set of db.
See: http://www.yacy-forum.de/viewtopic.php?p=11836#11836
http://www.yacy-forum.de/viewtopic.php?p=11814#11814
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@995 6c8d7289-2bf4-0310-a012-ef5d649a1542
2005-10-28 07:15:00 +00:00
orbiter
8d827cdb30
tried to fix problems with order of network list by last-seen (which could also improve the network picture)
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@980 6c8d7289-2bf4-0310-a012-ef5d649a1542
2005-10-24 14:07:43 +00:00
theli
446e7e8bef
*) Bugfix for Seed-Upload - Permission denied problem
...
See: http://www.yacy-forum.de/viewtopic.php?p=11648#11648
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@978 6c8d7289-2bf4-0310-a012-ef5d649a1542
2005-10-24 08:54:46 +00:00
orbiter
097009d910
experimental visualization of DHT access during global search (temporary)
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@977 6c8d7289-2bf4-0310-a012-ef5d649a1542
2005-10-24 00:34:15 +00:00
orbiter
4dcbc26ef1
introduction of search profiles; very experimental
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@976 6c8d7289-2bf4-0310-a012-ef5d649a1542
2005-10-23 17:50:27 +00:00
theli
9a5ab62928
*) Adding yacy specific X-YACY-Index-Control header which can be used by clients
...
to disallow yacy to index the response that belongs to the request where
X-YACY-Index-Contro is set to "no-index"
*) Bugfix for Seed-List download via Remote Proxy.
Now the pragma and cache-control http headers of the request are properly set to "no-cache"
See: http://www.yacy-forum.de/viewtopic.php?p=11639#11639
*) Bugfix for http-Proxy
yacy has ignored "no-cache"- pragma and cache-control http headers that were send in requests.
Now, these request headers are evaluated properly
TODO: Missing evaluation of "no-store" request headers
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@971 6c8d7289-2bf4-0310-a012-ef5d649a1542
2005-10-23 10:35:05 +00:00
theli
5a25ad9109
*) Bugfix for useRemoteProxy4YACY and useRemoteProxy4SSL check
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@969 6c8d7289-2bf4-0310-a012-ef5d649a1542
2005-10-23 08:59:11 +00:00
theli
02d9af1a70
*) Restructuring and extending of Remote Proxy Support
...
- remote proxy configuration can now be "really" changed on the fly and takes effect immediately
- adding possibility to disable remote proxy usage for yacy->yacy communication
- adding possibility to disable remote proxy usage for ssl
- restructuring proxy configuration so that it is stored in a single place now
*) Adding possibility to import a foreign word DB (or even more of them in parallel)
at runtime into the peers DB
- this can be done by calling IndexImport_p.html
- ATTENTION: please not that at the moment this thread must be aborted via gui
before a normal server shutdown is done.
- TODO: integrating IndexImport Thread into normal server shutdown
- TODO: Adding posibility to import crawl-queues, etc. from foreign peers
- TODO: removing old import function from yacy.java and calling the new routines instead
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@968 6c8d7289-2bf4-0310-a012-ef5d649a1542
2005-10-22 13:28:04 +00:00
theli
af3060938b
*) Bugfix for manual peer ping functionality
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@965 6c8d7289-2bf4-0310-a012-ef5d649a1542
2005-10-20 11:43:43 +00:00
theli
40777556c5
*) Connection Tracking
...
- adding automatic refresh
- accepts new parameter nameLookup which can be used to deactivate
yacy-peer name lookup (because we have problems with this on large seed-dbs)
*) ViewFile
New page that can be used to view
- original content
- plain text content
- parsed content
- parsed sentences
of a webpage specified by there url hash
Mainly for debugging purpose at the moment
*) Robots.txt
Bugfix for if-modified-since usage
TODO: synchronization of downloads to avoid loading the same robots-file
multiple times in parallel by different threads
*) Shutdown
Better abortion of transferRWI and transferURL sessions on server shutdown
*) Status Page
Adding icon to start/stop crawling via status page
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@950 6c8d7289-2bf4-0310-a012-ef5d649a1542
2005-10-18 07:45:27 +00:00
borg-0300
bcb0d6d5ff
changed setLastSeen(long rd) to setLastSeen();
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@949 6c8d7289-2bf4-0310-a012-ef5d649a1542
2005-10-17 23:23:17 +00:00
borg-0300
cdbaf637fb
added
...
- getIP(), getJunior(), getSenior(), getPrincipal();
- setIP(), setJunior(), setSenior(), setPrincipal(), setLastSeen(long rd);
- isPeerOK(), isOnline(String type);
next try to remove hello.class java.util.ConcurrentModificationException:null ;)
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@948 6c8d7289-2bf4-0310-a012-ef5d649a1542
2005-10-17 23:12:40 +00:00
borg-0300
e642a5d8b7
more constants
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@947 6c8d7289-2bf4-0310-a012-ef5d649a1542
2005-10-17 15:46:12 +00:00
borg-0300
d77b982083
small fix for last commit
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@944 6c8d7289-2bf4-0310-a012-ef5d649a1542
2005-10-16 00:15:23 +00:00
borg-0300
b00cd5640b
bugfix for 'hello.class java.util.ConcurrentModificationException:null'
...
finals
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@943 6c8d7289-2bf4-0310-a012-ef5d649a1542
2005-10-15 23:37:37 +00:00
orbiter
6260942590
changed search process: received indexes are now buffered and written to wordIndex after search
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@934 6c8d7289-2bf4-0310-a012-ef5d649a1542
2005-10-13 13:57:15 +00:00
orbiter
bc56a88cc8
further refactoring of search
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@925 6c8d7289-2bf4-0310-a012-ef5d649a1542
2005-10-13 00:05:30 +00:00
theli
c8a35a0130
*) Adding new connection tracking page (currently only for incoming connections)
...
*) Displaying statistic for incoming connections on status page
*) Bugfix for Loop-Access Bug when trying to access the yacy page while yacy is configured as proxy
See: http://www.yacy-forum.de/viewtopic.php?p=6826
*) Bugfix for Referer Bug
See: http://www.yacy-forum.de/viewtopic.php?p=11098#11098
*) Adding reverse Name lookup for yacy-domain names (used by the connection tracking page)
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@916 6c8d7289-2bf4-0310-a012-ef5d649a1542
2005-10-12 08:17:43 +00:00
orbiter
e85989510a
update to network image; added disconneced peers by disconnection time and changed colors
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@890 6c8d7289-2bf4-0310-a012-ef5d649a1542
2005-10-09 17:20:40 +00:00
orbiter
d666b61b83
fix for news-deletion, see also http://www.yacy-forum.de/viewtopic.php?p=11000#11000
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@885 6c8d7289-2bf4-0310-a012-ef5d649a1542
2005-10-09 14:31:35 +00:00
borg-0300
4180c422e8
cleaned, finals, Properties
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@884 6c8d7289-2bf4-0310-a012-ef5d649a1542
2005-10-09 13:18:48 +00:00
orbiter
c1c94111b0
added new network picture at Network menu using the new image-servlet method
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@880 6c8d7289-2bf4-0310-a012-ef5d649a1542
2005-10-08 00:37:43 +00:00
theli
a2fa75e688
*) Asynchronous queuing of crawl job URLs (stackCrawl)
...
various checks like the blacklist check or the robots.txt disallow check are now
done by a separate thread to unburden the indexer thread(s)
TODO: maybe we have to introduce a threadpool here if it turn out that this single
thread is a bottleneck because of the time consuming robots.txt downloads
*) improved index transfer
The index selection and transmission is done in parallel now to improve index
transfer performance.
TODO: maybe we could speed up performance by unsing multiple transmission threads in
parallel instead of only a single one.
*) gzip encoded post requests
it is now configureable if a gzip encoded post request should be send on
intex transfer/distribution
*) storage Peer (very experimentell and not optimized yet)
Now it's possible to send the result of the yacy indexer thread to a remote peer
istead of storing the indexed words locally.
This could be done by setting the property "storagePeerHash" in the yacy config file
- Please note that if the index transfer fails, the index ist stored locally.
- TODO: currently this index transfer is done by the indexer thread.
To seedup the indexer
a) this transmission should be done in parallel and
b) multiple chunks should be bundled and transfered together
*) general performance improvements
- better memory cleanup after http request processing has finished
- replacing some string concatenations with stringBuffers
- replacing BufferedInputStreams with serverByteBuffer
- replacing vectors with arraylists wherever possible
- replacing hashtables with hashmaps wherever possible
This was done because function calls to verctor or hashtable functions
take 3 time longer than calls to functions of arraylists or hashmaps.
TODO: we should take a look on the class serverObject which is inherited from hashmap
Do we realy need a synchronization for this class?
TODO: replace arraylists with linkedLists if random access to the list elements is not needed
*) Robots Parser supports if-modified-since downloads now
If the downloaded robots.txt file is older than 7 days the robots parser tries to
download the robots.txt with the if-modified-since header to avoid unnecessary downloads
if the file was not changed. Additionally the ETag header is used to detect changes.
*) Crawler: better handling of unsupported mimeTypes + FileExtension
*) Bugfix: plasmaWordIndexEntity was not closed correctly in
- query.java
- plasmaswitchboard.java
*) function minimizeUrlDB added to yacy.java
this function tests the current urlHashDB for unused urls
ATTENTION: please don't use this function at the moment because
it causes the wordIndexDB to flush all words into the
word directory!
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@853 6c8d7289-2bf4-0310-a012-ef5d649a1542
2005-10-05 10:45:33 +00:00
orbiter
e5f8163203
fixed a bug with news; news moving could lead to shurtcut loop / 100% CPU; appeared when clicked on a 'Profile' news in Network menu
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@845 6c8d7289-2bf4-0310-a012-ef5d649a1542
2005-10-03 23:33:10 +00:00
orbiter
fbb5e36b80
documentation update
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@843 6c8d7289-2bf4-0310-a012-ef5d649a1542
2005-10-03 23:04:11 +00:00
orbiter
0054d3b1a6
added age in network menu
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@809 6c8d7289-2bf4-0310-a012-ef5d649a1542
2005-09-28 22:54:31 +00:00
orbiter
7fc822a59b
changed handling of time-zones
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@801 6c8d7289-2bf4-0310-a012-ef5d649a1542
2005-09-27 16:28:55 +00:00
theli
70a5681a4f
*) Bugfix for inactive scp seed uploader
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@779 6c8d7289-2bf4-0310-a012-ef5d649a1542
2005-09-22 23:04:26 +00:00
orbiter
495bc8bec6
removed cache-control from low and medium priority caches which reduces memory use and computation overhead
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@774 6c8d7289-2bf4-0310-a012-ef5d649a1542
2005-09-22 20:01:26 +00:00
theli
1dc94e7753
*) Adding support for gzip content-encoding of http post requests
...
used to transferRWIs and transferURLs.
See: http://www.yacy-forum.de/viewtopic.php?t=1167#10020
*) adding yacyVersion.java containing constants defining yacy versions
that support a given feature.
Needed to determine if a remote peer is able to decode gzip
content-encoded http post bodies properly.
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@772 6c8d7289-2bf4-0310-a012-ef5d649a1542
2005-09-22 10:30:55 +00:00
borg-0300
a1f5027a88
finals;
...
cleaned;
Properties;
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@770 6c8d7289-2bf4-0310-a012-ef5d649a1542
2005-09-21 23:18:12 +00:00
borg-0300
1dd7047af5
finals;
...
cleaned;
Properties;
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@767 6c8d7289-2bf4-0310-a012-ef5d649a1542
2005-09-21 21:32:43 +00:00
orbiter
96a5b6e8fb
removed yacy peer types from serverSwitch
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@758 6c8d7289-2bf4-0310-a012-ef5d649a1542
2005-09-20 23:15:33 +00:00
theli
b990dc1ad1
*) Replacing jsch 0.1.19 lib with newer version 0.1.21
...
*) Replacing PDFBox 0.7.1 lib with newer version 0.7.2
*) Refactoring of classes httpd/httpc/httpHeaders to
make many methods for httpHeader/Requestline parsing
reusable for new icap implementation
*) adding chunked input stream support
- needed by new icap implementation
- needed by future httpc HTTP/1.1 support
*) httpd.java
- moving all connection property contants to class httpHeader
- moving readHeader function to class httpHeader
- moving parseQuery function to class httpHeader
- moving handleTransparentProxy function to class httpHeader
*) httpHeader.java
- adding new fuction to parse the http response line
- adding new function to converte http headers to a string that
can be send to the client
- adding a function that generates a proper url using all parsed
connection properties
*) ICAP Support
- yacy now supports handling of icap response modification requests
- this feature can be used by other icap enabled proxies to contact
yacy as icap server, and to handover the downloaded content to yacy.logging
for indexing
- functionality was successfully tested with squid 2.5Stable 10 + icap patch
- further icap services e.g. URL filtering based on yacy's blacklists are possible
*) plasmaSwitchboard.java
- htcache entries that are still needed for indexing are now properly registered
as in use after system restart
- extended logging: log message now shows parsing and indexing time for each sb. entry
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@757 6c8d7289-2bf4-0310-a012-ef5d649a1542
2005-09-20 21:49:47 +00:00
orbiter
af9021e956
fixed bug with news caching
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@754 6c8d7289-2bf4-0310-a012-ef5d649a1542
2005-09-20 11:30:03 +00:00
orbiter
71a31f0902
integrated and extended new memory performance menu; found and fixed bug in DHT caching
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@752 6c8d7289-2bf4-0310-a012-ef5d649a1542
2005-09-20 10:54:20 +00:00
orbiter
fb52a82008
added new performance page for memory settings
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@751 6c8d7289-2bf4-0310-a012-ef5d649a1542
2005-09-20 10:10:34 +00:00
orbiter
50a9500035
fixed 100% CPU bug with news queue deletion
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@735 6c8d7289-2bf4-0310-a012-ef5d649a1542
2005-09-15 22:51:41 +00:00
orbiter
2148c0cf49
replaced kelondro storage core; much less objects in kelondro cache now; less IO from DB
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@724 6c8d7289-2bf4-0310-a012-ef5d649a1542
2005-09-14 10:10:49 +00:00
theli
dff96601fe
*) Bugfix for transferURL:
...
URL list index was not incremented properly.
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@723 6c8d7289-2bf4-0310-a012-ef5d649a1542
2005-09-14 07:20:14 +00:00
rramthun
10e7d6f02b
Bugfix for http://www.yacy-forum.de/viewtopic.php?t=1053
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@713 6c8d7289-2bf4-0310-a012-ef5d649a1542
2005-09-12 15:33:51 +00:00
theli
2cb084d426
*) Complete Index Transfer
...
See: http://www.yacy-forum.de/viewtopic.php?p=9622
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@707 6c8d7289-2bf4-0310-a012-ef5d649a1542
2005-09-12 10:37:16 +00:00
borg-0300
801e902795
small change
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@698 6c8d7289-2bf4-0310-a012-ef5d649a1542
2005-09-10 17:47:48 +00:00
theli
019cc716db
*) Undoing last changes on yacySeed. Seems not to work properly.
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@697 6c8d7289-2bf4-0310-a012-ef5d649a1542
2005-09-10 17:46:17 +00:00
theli
177e8af5b7
*) Bugfix for ConcurrentModification in kelondroAbstractRA.writeMap caused by yacySeed.getMap()
...
See: http://www.yacy-forum.de/viewtopic.php?p=9523
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@695 6c8d7289-2bf4-0310-a012-ef5d649a1542
2005-09-10 05:59:12 +00:00
theli
d3c923e6b9
*) Bugfix for "ConcurrentModificationException in hello.class"
...
See: http://www.yacy-forum.de/viewtopic.php?t=723
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@694 6c8d7289-2bf4-0310-a012-ef5d649a1542
2005-09-10 05:42:30 +00:00
rramthun
02c242ae22
minor changes
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@693 6c8d7289-2bf4-0310-a012-ef5d649a1542
2005-09-09 21:01:53 +00:00
rramthun
7c86c36210
undoing one part of the last commit. do not know, why it didn't work...
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@681 6c8d7289-2bf4-0310-a012-ef5d649a1542
2005-09-07 17:40:53 +00:00
rramthun
a79913c6ea
updated german language file
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@680 6c8d7289-2bf4-0310-a012-ef5d649a1542
2005-09-07 17:21:49 +00:00
borg-0300
718950c5da
small change
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@679 6c8d7289-2bf4-0310-a012-ef5d649a1542
2005-09-07 15:20:12 +00:00
theli
a7256e8f4e
*) Adding X-Forwarded-For Header
...
See: http://www.yacy-forum.de/viewtopic.php?t=1118&highlight=xforwardedfor
*) httpc.java: Bugfix for incorrect http response statuscode parsing
In some situations the statustext whas chopped
*) Adding a lot of fileheaders containing YaCy copyright and license
*) httpd.java: Adding additional debugging http header that should help du detect
the "binary data in browser window" bug.
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@653 6c8d7289-2bf4-0310-a012-ef5d649a1542
2005-09-05 08:01:54 +00:00
theli
4fd5b95b1f
*) Renaming Logger function names to reflect the proper Java Logging API Loglevels
...
- please use logFine instead of logDebug
- please use logSevere instead of logFailure and logError
See: http://www.yacy-forum.de/viewtopic.php?p=8726#8726
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@615 6c8d7289-2bf4-0310-a012-ef5d649a1542
2005-08-30 21:32:59 +00:00
theli
6adf8a4bde
*) Renaming Logger function names to reflect the proper Java Logging API Loglevels
...
- please use logFine instead of logDebug
- please use logFailure instead of logError
See: http://www.yacy-forum.de/viewtopic.php?p=8726#8726
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@614 6c8d7289-2bf4-0310-a012-ef5d649a1542
2005-08-30 21:10:39 +00:00
theli
3dfda1c9da
*) More verbose output on ftp-seed-upload failure
...
See: http://www.yacy-forum.de/viewtopic.php?p=8000#8000
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@605 6c8d7289-2bf4-0310-a012-ef5d649a1542
2005-08-30 12:18:41 +00:00
theli
a812fb86cc
*) Port Forwarding Feature does not detect broken connection properly.
...
Therefor a test-request was added to the isConnected function to detect broken connections
and to keep open connections alive
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@596 6c8d7289-2bf4-0310-a012-ef5d649a1542
2005-08-29 11:39:10 +00:00
allo
66ebce1109
use staticIP more often
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@592 6c8d7289-2bf4-0310-a012-ef5d649a1542
2005-08-28 16:55:52 +00:00
allo
eb6365c069
local Bootstrapping bug.
...
use yacyDebugMode=true to allow local bootstrapping
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@572 6c8d7289-2bf4-0310-a012-ef5d649a1542
2005-08-22 12:13:19 +00:00
rramthun
26289e955c
Updates of the language file
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@563 6c8d7289-2bf4-0310-a012-ef5d649a1542
2005-08-20 12:09:27 +00:00
theli
0d0bc02f38
*) Printing stacktrace into log on seed-upload error
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@562 6c8d7289-2bf4-0310-a012-ef5d649a1542
2005-08-19 22:59:33 +00:00
theli
115c4edfcc
*) Adding additional logging statements to help debugging seed-upload problems
...
See: http://www.yacy-forum.de/viewtopic.php?t=975&highlight=
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@561 6c8d7289-2bf4-0310-a012-ef5d649a1542
2005-08-19 09:32:30 +00:00
orbiter
25f632dbd9
more DHT bugfixes and better logging of DHT effects
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@542 6c8d7289-2bf4-0310-a012-ef5d649a1542
2005-08-16 00:31:15 +00:00
orbiter
5cb00889d9
enhancements to dht selection, search and search presentation
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@540 6c8d7289-2bf4-0310-a012-ef5d649a1542
2005-08-15 01:12:25 +00:00
orbiter
ba0a486328
moved printStackTrace() to logging
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@539 6c8d7289-2bf4-0310-a012-ef5d649a1542
2005-08-14 23:35:18 +00:00
orbiter
cd10370992
several bugfixes and dht selection / logging improvement
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@531 6c8d7289-2bf4-0310-a012-ef5d649a1542
2005-08-14 00:57:30 +00:00
orbiter
c8a7a85ce2
fix for http://www.yacy-forum.de/viewtopic.php?p=7384#7384
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@529 6c8d7289-2bf4-0310-a012-ef5d649a1542
2005-08-13 21:41:22 +00:00
orbiter
98b028038c
bugfix, see http://www.yacy-forum.de/viewtopic.php?p=7370#7370
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@527 6c8d7289-2bf4-0310-a012-ef5d649a1542
2005-08-12 23:30:42 +00:00
orbiter
7db543a9fa
fixes for several dht misbehaviours
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@524 6c8d7289-2bf4-0310-a012-ef5d649a1542
2005-08-12 22:14:24 +00:00
orbiter
5716f8521d
bug fixes for word ordering and dht index selection
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@521 6c8d7289-2bf4-0310-a012-ef5d649a1542
2005-08-12 14:06:47 +00:00
orbiter
248c24b60a
intermission-feature usage in case of local and remote search
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@510 6c8d7289-2bf4-0310-a012-ef5d649a1542
2005-08-09 20:43:37 +00:00
orbiter
ec07928807
faster listing and news aging enhancement
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@505 6c8d7289-2bf4-0310-a012-ef5d649a1542
2005-08-08 18:05:43 +00:00
rramthun
1eadbc6b91
Modified build.xml to include Win9x scripts
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@501 6c8d7289-2bf4-0310-a012-ef5d649a1542
2005-08-07 09:26:23 +00:00
orbiter
2dd2533e6d
cut for 0.394
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@489 6c8d7289-2bf4-0310-a012-ef5d649a1542
2005-08-03 14:28:28 +00:00
orbiter
2d8557cb10
minor changes
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@487 6c8d7289-2bf4-0310-a012-ef5d649a1542
2005-08-03 02:02:39 +00:00
orbiter
91163db52e
fix for more time-related problems in proxy
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@486 6c8d7289-2bf4-0310-a012-ef5d649a1542
2005-08-03 00:52:32 +00:00
orbiter
fb6f238d70
fix for expires-problem
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@485 6c8d7289-2bf4-0310-a012-ef5d649a1542
2005-08-03 00:28:12 +00:00
orbiter
40da910f41
bugfixes and automatic news-cleanup
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@481 6c8d7289-2bf4-0310-a012-ef5d649a1542
2005-08-02 16:03:35 +00:00
theli
228b04b499
*) Bugfix for "wrong seed-upload timestamp" problem
...
http://www.yacy-forum.de/viewtopic.php?t=817
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@480 6c8d7289-2bf4-0310-a012-ef5d649a1542
2005-08-02 15:36:10 +00:00
orbiter
e84a177c49
many bigfixes
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@475 6c8d7289-2bf4-0310-a012-ef5d649a1542
2005-08-02 02:18:01 +00:00
orbiter
9ee8a5ba6c
fixed big in yacynews
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@474 6c8d7289-2bf4-0310-a012-ef5d649a1542
2005-08-02 00:50:20 +00:00
orbiter
bcc292ad12
added self-news; added news in case of wiki update; added crawl announcement on Network page
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@469 6c8d7289-2bf4-0310-a012-ef5d649a1542
2005-08-01 12:47:47 +00:00
orbiter
d34eb23e4e
fixed news; added news appearance on Network and IndexCreate page; added intention string to global crawl
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@466 6c8d7289-2bf4-0310-a012-ef5d649a1542
2005-08-01 01:12:02 +00:00
orbiter
5672709ef3
several bugfixes for YaCyNews
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@465 6c8d7289-2bf4-0310-a012-ef5d649a1542
2005-07-31 14:59:31 +00:00
orbiter
e24dbde217
better logging for WRONG seed
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@463 6c8d7289-2bf4-0310-a012-ef5d649a1542
2005-07-31 11:11:29 +00:00
orbiter
1022fbeb65
many YaCyNews fixes
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@461 6c8d7289-2bf4-0310-a012-ef5d649a1542
2005-07-31 01:54:46 +00:00
orbiter
13abd8b6e7
added news-creation at crawl start
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@460 6c8d7289-2bf4-0310-a012-ef5d649a1542
2005-07-30 11:57:19 +00:00
orbiter
849b194149
fixed news receipt and added processing buttons on News page
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@458 6c8d7289-2bf4-0310-a012-ef5d649a1542
2005-07-30 07:15:39 +00:00
orbiter
84b74d40f3
yacyNews bugfix
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@457 6c8d7289-2bf4-0310-a012-ef5d649a1542
2005-07-29 15:29:06 +00:00
orbiter
60117a2073
integrated YaCyNews basis
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@455 6c8d7289-2bf4-0310-a012-ef5d649a1542
2005-07-29 14:10:19 +00:00
orbiter
b8e294f149
corrected wrong default peer-names
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@454 6c8d7289-2bf4-0310-a012-ef5d649a1542
2005-07-29 08:48:36 +00:00
orbiter
f32d8bdfa8
fixed DHT cache memory assignment
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@452 6c8d7289-2bf4-0310-a012-ef5d649a1542
2005-07-28 18:22:02 +00:00
orbiter
f663f26cfd
catch of another IOException
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@451 6c8d7289-2bf4-0310-a012-ef5d649a1542
2005-07-28 17:03:13 +00:00
orbiter
85877413a0
tried to fix principal bug .. not succeeded
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@440 6c8d7289-2bf4-0310-a012-ef5d649a1542
2005-07-27 13:38:46 +00:00
rramthun
bf7d5d1310
Usage of logging-class
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@432 6c8d7289-2bf4-0310-a012-ef5d649a1542
2005-07-25 14:46:36 +00:00
orbiter
af67c633d5
doc-changes and more strict brute-force handling
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@431 6c8d7289-2bf4-0310-a012-ef5d649a1542
2005-07-25 09:56:54 +00:00