rramthun
157d01cc97
Checked in old version, sorry.
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1954 6c8d7289-2bf4-0310-a012-ef5d649a1542
2006-03-23 20:16:02 +00:00
rramthun
42b0b10a95
-Adding Windows Media to types which are not sended compressed
...
-Renaming writeandzip to writeandgzip to avoid confusion about type of compression
-Adding new startup message to windows script
-The usual language "enhancements" ;-)
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1953 6c8d7289-2bf4-0310-a012-ef5d649a1542
2006-03-23 20:12:23 +00:00
allo
2ed4fa96b7
tagCache
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1952 6c8d7289-2bf4-0310-a012-ef5d649a1542
2006-03-23 19:04:05 +00:00
allo
330eb9c74f
bookmarkDB cleanup
...
(preparation for tagCache)
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1951 6c8d7289-2bf4-0310-a012-ef5d649a1542
2006-03-23 17:56:32 +00:00
orbiter
e82899ba57
fixed missing urls map initializer
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1950 6c8d7289-2bf4-0310-a012-ef5d649a1542
2006-03-23 16:13:23 +00:00
orbiter
63f39ac7b5
added 3 new crawling steering options:
...
- re-crawl by age of page (enter in minutes)
- auto-domain-filter
- maximum number of pages per domain
NOT YET TESTED!
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1949 6c8d7289-2bf4-0310-a012-ef5d649a1542
2006-03-23 16:05:16 +00:00
orbiter
1fc3b34be6
some pre-work (without function yet) to implement:
...
- re-crawl (by age of last crawl)
- auto-crawl-filter by crawl depth (to be explained..)
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1948 6c8d7289-2bf4-0310-a012-ef5d649a1542
2006-03-22 15:28:17 +00:00
orbiter
0a4c2e89ed
remote crawl orders are now only accepted if sum over all
...
queues is less than 100 (the indexing queue was not measured before)
see also: http://www.yacy-forum.de/viewtopic.php?p=19374#19374
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1947 6c8d7289-2bf4-0310-a012-ef5d649a1542
2006-03-22 14:23:24 +00:00
theli
c9e6b5e391
*) check size of indexing-queue and crawler pool before processing remote triggered crawl jobs
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1946 6c8d7289-2bf4-0310-a012-ef5d649a1542
2006-03-22 14:19:03 +00:00
orbiter
1509314ea6
set tighter control during DHT index and peer selection
...
see http://www.yacy-forum.de/viewtopic.php?p=19329#19329
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1945 6c8d7289-2bf4-0310-a012-ef5d649a1542
2006-03-22 13:54:35 +00:00
hydrox
fcc0683200
*) undoing last commit
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1944 6c8d7289-2bf4-0310-a012-ef5d649a1542
2006-03-22 09:01:25 +00:00
hydrox
9411961eec
*) another little fix for DHT-Transfer
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1943 6c8d7289-2bf4-0310-a012-ef5d649a1542
2006-03-22 08:49:39 +00:00
borg-0300
77f3237de3
adapted for isListed()
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1942 6c8d7289-2bf4-0310-a012-ef5d649a1542
2006-03-21 20:55:59 +00:00
hydrox
8b14a0c833
*) little fix for DHT-Transfer
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1941 6c8d7289-2bf4-0310-a012-ef5d649a1542
2006-03-21 10:35:57 +00:00
orbiter
1f4412a146
adopted isListed to discussed new behavior as discussed (url, getFile)
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1940 6c8d7289-2bf4-0310-a012-ef5d649a1542
2006-03-20 22:31:59 +00:00
orbiter
59fc55ea1e
added checks to protect peers from wrong seeds
...
see also: http://www.yacy-forum.de/viewtopic.php?p=19249#19249
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1939 6c8d7289-2bf4-0310-a012-ef5d649a1542
2006-03-20 20:31:51 +00:00
allo
eba22e7f89
include the defaultskins in the Release.
...
at least skins/default.css is needed.
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1938 6c8d7289-2bf4-0310-a012-ef5d649a1542
2006-03-20 12:17:22 +00:00
orbiter
cded4923ff
changed start-up message and Mac startup method
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1937 6c8d7289-2bf4-0310-a012-ef5d649a1542
2006-03-19 23:25:54 +00:00
orbiter
063ef4660a
bug?
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1936 6c8d7289-2bf4-0310-a012-ef5d649a1542
2006-03-19 22:06:15 +00:00
orbiter
82358677a9
added another shiftK2W to flushCacheSome
...
this should fix the bug that the DHT cache is not flushed if there is no indexing
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1935 6c8d7289-2bf4-0310-a012-ef5d649a1542
2006-03-19 21:33:31 +00:00
borg-0300
cb23fc3d83
keywords added
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1934 6c8d7289-2bf4-0310-a012-ef5d649a1542
2006-03-19 12:36:39 +00:00
borg-0300
1258df8133
no "[0-9]+" in topwords
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1933 6c8d7289-2bf4-0310-a012-ef5d649a1542
2006-03-19 12:33:50 +00:00
orbiter
128e4ab199
- in serverSystem: maxPathLength is now a variable, not a method
...
- upon startup the calculated maximum path length is shown
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1932 6c8d7289-2bf4-0310-a012-ef5d649a1542
2006-03-19 01:33:20 +00:00
orbiter
a37b09e303
implemented automatic adoption of chunk-read-ahead in kelondroTree to needed chunk size
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1931 6c8d7289-2bf4-0310-a012-ef5d649a1542
2006-03-19 01:27:33 +00:00
orbiter
30e3e3a0fd
adopted MAXPATHLENGTH to host system capabilities
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1930 6c8d7289-2bf4-0310-a012-ef5d649a1542
2006-03-19 00:29:27 +00:00
borg-0300
d808765087
something Javadoc
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1929 6c8d7289-2bf4-0310-a012-ef5d649a1542
2006-03-18 21:47:04 +00:00
borg-0300
85bb8e32a1
Bugfix for last commit
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1928 6c8d7289-2bf4-0310-a012-ef5d649a1542
2006-03-18 19:05:56 +00:00
borg-0300
3fe402069f
try to fix
...
see: http://www.yacy-forum.de/viewtopic.php?p=19175#19175
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1927 6c8d7289-2bf4-0310-a012-ef5d649a1542
2006-03-18 18:51:52 +00:00
orbiter
f16f1f15cd
bugfix for 100% CPU bug; thanks to Matthias for analysis
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1926 6c8d7289-2bf4-0310-a012-ef5d649a1542
2006-03-18 16:14:24 +00:00
borg-0300
254a13efd9
MAXPATHLENGTH used
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1925 6c8d7289-2bf4-0310-a012-ef5d649a1542
2006-03-18 15:20:50 +00:00
allo
2635244489
Bugfix from cato
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1924 6c8d7289-2bf4-0310-a012-ef5d649a1542
2006-03-18 14:08:34 +00:00
borg-0300
8865948e4e
Cleanup;
...
Methode replaceRegex added;
Constant MAXPATHLENGTH added;
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1923 6c8d7289-2bf4-0310-a012-ef5d649a1542
2006-03-18 13:34:32 +00:00
rramthun
9c85820d35
added MIME-type for wmv and rm
...
removed double copyright at startup
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1922 6c8d7289-2bf4-0310-a012-ef5d649a1542
2006-03-18 12:39:00 +00:00
orbiter
6c70f4a0cf
renamed wordHashes for a word hash set generation to wordHashSet
...
This was done because the wordHashes iterator will get another integer
parameter and then conflicts with the wordHashes set generation
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1921 6c8d7289-2bf4-0310-a012-ef5d649a1542
2006-03-18 01:04:16 +00:00
orbiter
d5f8f40c31
removed correcting iterator
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1920 6c8d7289-2bf4-0310-a012-ef5d649a1542
2006-03-18 01:01:00 +00:00
orbiter
526407f32e
adoptions, fixes for last commit
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1919 6c8d7289-2bf4-0310-a012-ef5d649a1542
2006-03-18 00:21:34 +00:00
orbiter
488a0ed580
replaced old keyIterator and rowIterator by buffered iterators
...
that are synchronized with database access
Main change is done in kelondroTree, other classes are only adoptions
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1918 6c8d7289-2bf4-0310-a012-ef5d649a1542
2006-03-17 23:43:24 +00:00
hermens
4e9a8f41fd
rwiDBCleaner + dbImporter: Iterate over small excerpts of
...
word hashes instead of the whole DB especially while changing
the DB in the process.
see http://www.yacy-forum.de/viewtopic.php?p=19136#19136
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1917 6c8d7289-2bf4-0310-a012-ef5d649a1542
2006-03-17 23:39:10 +00:00
hermens
474379ae63
remove TABs from plasmaDbImporter.java
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1916 6c8d7289-2bf4-0310-a012-ef5d649a1542
2006-03-17 21:52:36 +00:00
allo
2b31f51896
bugfix for last commit
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1915 6c8d7289-2bf4-0310-a012-ef5d649a1542
2006-03-17 21:09:07 +00:00
orbiter
dba02f399f
starting of re-design of kelondroTree iterator
...
- new access to iterator
- added many IOException handling in other Classes
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1914 6c8d7289-2bf4-0310-a012-ef5d649a1542
2006-03-17 20:52:43 +00:00
allo
d2f34a2e94
skin in DATA/HTDOCS (works after upgrade, works when only DATA is writable).
...
preference of skinfiles:
-style.css will be overwritten on every new SVN, if "DATA/SKINS/"+currentSkin+".css" exists.
-DATA/SKINS will be overwritten with defaultskins from /skins
so develop your skins in DATA/SKINS, and put them into skins for distribution.
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1913 6c8d7289-2bf4-0310-a012-ef5d649a1542
2006-03-17 20:48:00 +00:00
allo
59b9540eb6
save the current Skin
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1912 6c8d7289-2bf4-0310-a012-ef5d649a1542
2006-03-17 20:11:16 +00:00
allo
3c038727a4
x-chat script from cato
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1911 6c8d7289-2bf4-0310-a012-ef5d649a1542
2006-03-17 19:55:03 +00:00
orbiter
f02b426073
made kelondroTree.nodeIterator private
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1910 6c8d7289-2bf4-0310-a012-ef5d649a1542
2006-03-17 18:10:48 +00:00
borg-0300
5f6fdf1786
Bugfix for getCachePath(URL url)
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1909 6c8d7289-2bf4-0310-a012-ef5d649a1542
2006-03-17 16:39:29 +00:00
orbiter
303b6463a8
added debug line to URL storage for testing
...
see http://www.yacy-forum.de/viewtopic.php?p=19129#19129
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1908 6c8d7289-2bf4-0310-a012-ef5d649a1542
2006-03-17 16:30:31 +00:00
orbiter
431a4f3609
eliminated correcting iterator in kelondroTree
...
VERY EXPERIMENTAL! NOT TESTED!
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1907 6c8d7289-2bf4-0310-a012-ef5d649a1542
2006-03-17 14:20:04 +00:00
orbiter
91dca2cd8d
fixed a bug in last commit: LURL entries cannot be written,
...
because a stored property was not set to false (but true)
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1906 6c8d7289-2bf4-0310-a012-ef5d649a1542
2006-03-17 13:07:34 +00:00
orbiter
3286b1f498
re-organisation of lurl-creation and -stacking
...
this was necessary to prevent useless write to the database
in case of blacklist appearance of the url
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1905 6c8d7289-2bf4-0310-a012-ef5d649a1542
2006-03-17 10:16:07 +00:00