Marown/yacy_search_server
1
0
Fork 0
mirror of https://github.com/yacy/yacy_search_server.git synced 2024-09-21 00:00:13 +02:00
Code Issues Actions Packages Projects Releases Wiki Activity
2,120 Commits 1 Branch 20 Tags 836 MiB
761d3bba54
Commit Graph

1381 Commits

Author SHA1 Message Date
auron_x
586b256627 *)fixed Blog-bug with german Umlaute 
*)corrected wrong html in blog
see http://www.yacy-forum.de/viewtopic.php?t=2330

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2083 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2006-05-11 14:03:15 +00:00
allo
9938c252dd better Errorhandling for proxyAccounts 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2082 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2006-05-11 13:12:35 +00:00
orbiter
35995cf8c7 some enhancements to caching and kelondroRA-methods 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2081 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2006-05-10 21:36:44 +00:00
orbiter
2e89477f22 undoing last change 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2080 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2006-05-10 16:48:49 +00:00
orbiter
a3a4525931 usage of a buffer to prevent too many cache lookups in kelondroMap 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2079 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2006-05-10 16:41:42 +00:00
orbiter
015d044c25 tried to fix some problems with latest changes to httpc 
very experimental!

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2078 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2006-05-10 16:01:14 +00:00
orbiter
55c5b41bd0 modified kelondroDyn to work better with new object caches 
(removed own single object cache)

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2077 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2006-05-10 13:57:31 +00:00
orbiter
26e3216bcc update to profile fetch behavior 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2076 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2006-05-10 09:37:18 +00:00
orbiter
3e31820c3d - corrections to PerformanceMemory display of object cache 
- configuration of object cache size in kelondroTree initializer

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2075 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2006-05-10 09:08:42 +00:00
orbiter
fd7c17e624 added virtual host support: 
all yacy-to-yacy communication now send the <peer-hexhash>.yacyh
virtual domain inside the http 'Host' property field.
This shall enable running a yacy peer on a virtual host.

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2074 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2006-05-09 13:11:00 +00:00
orbiter
461548698c configuration of index transfer chunk size 
see http://www.yacy-forum.de/viewtopic.php?p=20951#20951
new properties in yacy.init:
indexDistribution.minChunkSize = 5
indexDistribution.maxChunkSize = 1000
indexDistribution.startChunkSize = 50

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2073 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2006-05-09 11:43:10 +00:00
orbiter
29b1b0823c added monitoring of new object cache to performanceMemory page 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2072 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2006-05-09 10:03:12 +00:00
orbiter
23ced30e83 fixed a bug in new caching 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2071 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2006-05-08 21:29:28 +00:00
low012
35f9418117 Fixed links and images. (http://www.yacy-forum.de/viewtopic.php?p=20908#20908) 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2070 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2006-05-08 14:23:22 +00:00
orbiter
bab9ee38e7 added very experimental object cache for all kelondroTree 
databases. This cache now takes up 10% of all cache sizes,
90% goes to the node routing cache we had so far.

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2069 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2006-05-08 13:37:27 +00:00
theli
9104001e7c *) Better error handling for assortment import 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2067 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2006-05-08 07:58:22 +00:00
low012
d40ccc5208 *) Partly undoing last commit\n*) Fixed bug that caused attributes of cells in tables not to be recognized as such. 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2066 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2006-05-07 22:53:17 +00:00
low012
adcad5eae6 *) Fixed a bug that prevented the usage of tables in the wiki. 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2065 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2006-05-07 18:15:00 +00:00
hermens
51e3bb576f Don't increase dhtTransferIndexCount when the last transferred index was smaller 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2064 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2006-05-07 17:44:33 +00:00
low012
152745b145 *) Added some characters and their HTML-codes. 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2060 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2006-05-06 00:23:29 +00:00
hermens
a0ca4c5fb8 Remove a possible race condition between DHT transfer and deQueue 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2059 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2006-05-05 13:17:00 +00:00
hermens
0cfba8950f Removing unnecessary and possibly dangerous synchronization of the wordIndex 
when deleting transferred indexes



git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2058 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2006-05-05 12:48:17 +00:00
orbiter
fe4ad214f1 configuration of time-out for fetching seed-lists during bootstraping 
(weil das beim linuxtag nur auf meinem Notebook nicht funtioniert)

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2054 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2006-05-03 21:27:30 +00:00
allo
1179c9ffff fix for the bookmark error. 
fixes: http://www.yacy-forum.de/viewtopic.php?t=2274

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2043 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2006-04-28 17:50:05 +00:00
orbiter
d6213f8a85 quickfix for http://www.yacy-forum.de/viewtopic.php?p=19482#19482 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2042 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2006-04-25 15:35:25 +00:00
orbiter
e9aadb2e11 quickfix for http://www.yacy-forum.de/viewtopic.php?p=19765#19765 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2041 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2006-04-25 15:21:17 +00:00
theli
727aac4768 *) Bugfix for Transparent-Proxy-Support <-> Port Forwarding problem 
See: http://www.yacy-forum.de/viewtopic.php?p=20358

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2039 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2006-04-25 05:29:20 +00:00
theli
cd4aeffea2 *) Bugfix: httpdFileHandler.java did not handle filenames with encoded chars correctly 
See: http://www.yacy-forum.de/viewtopic.php?t=2265

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2036 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2006-04-23 11:01:31 +00:00
theli
a10072b5c0 *) Adding timeout to ftpc.java 
See: http://www.yacy-forum.de/viewtopic.php?t=2268
   

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2035 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2006-04-23 10:35:46 +00:00
theli
76ea16a6cb *) Removing Keep-Alive header (is also a hopByHop header) 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2034 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2006-04-22 15:00:35 +00:00
orbiter
b0036249c1 added some attributes to network picture 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2032 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2006-04-21 21:21:35 +00:00
hermens
cbcf7418ef Cleanup synchronization in plasmaWordIndex 
-  only synchronize when changing data in more than one database
see: http://www.yacy-forum.de/viewtopic.php?t=2167



git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2031 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2006-04-20 14:00:47 +00:00
orbiter
60e5aff9fc some enhancements to the remote crawl trigger 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2030 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2006-04-20 11:53:15 +00:00
orbiter
dbe96e6541 added hand-over of search filter and prefer ranking to yacy protocol 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2029 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2006-04-20 10:15:00 +00:00
rramthun
0604203bce Updated and corrected German language file 
Changed Italian language file for an Italian/English interface and not Italian/German

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2024 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2006-04-18 11:37:03 +00:00
low012
4639c0ac6d *) cleaned up code 
*) fixed a bug where a line like <pre><pre>text</pre></pre><pre>text</pre> caused an exception
*) fixed a bug where a line like [=[=text=]=][=text=] caused an exception


git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2023 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2006-04-17 19:37:09 +00:00
orbiter
00a5d435e2 - fixed some bugs with domain filter 
- added new ranking filter "prefermask": urls that match the filter are ranked better


git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2022 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2006-04-13 23:19:36 +00:00
orbiter
14d6e476c9 tried to solve some problems with new picture viewer 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2019 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2006-04-10 22:34:47 +00:00
low012
23cdf319be *) cleaning up code 
*) fixed a bug concerning [= =] and <pre></pre> in tables


git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2018 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2006-04-10 22:04:52 +00:00
orbiter
9324425165 fix for remote crawl reject 
see http://www.yacy-forum.de/viewtopic.php?p=20075#20075

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2017 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2006-04-10 21:34:17 +00:00
borg-0300
30e4fc39a5 HTCache extended 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2015 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2006-04-08 13:05:04 +00:00
orbiter
d0dd8b14d2 fixed picture tag and presentation 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2014 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2006-04-07 22:09:59 +00:00
borg-0300
da6a8bafa2 rename currCacheSize -> curCacheSize; 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2010 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2006-04-07 13:15:15 +00:00
borg-0300
92110aea32 nullpointer fix for profile(); other minor change; 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2009 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2006-04-07 12:43:59 +00:00
orbiter
f0833b0328 introduced simple search interface 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2007 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2006-04-06 21:48:24 +00:00
orbiter
bd283b8443 fixed bugs: 
- null pointer exception during startup of a robinson-configured peer
- wrong time calculation of default value of re-crawl option

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2005 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2006-04-06 16:28:28 +00:00
orbiter
47b541b2d1 added better option handling in yacysearch 
added depth option for image presentation

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2001 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2006-04-05 10:34:24 +00:00
orbiter
c9e16bfd48 first try to insert image search (does not work yet) 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2000 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2006-04-04 23:12:10 +00:00
orbiter
f77775220b fixed parser error 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1999 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2006-04-04 22:28:46 +00:00
auron_x
0bf2eb4813 *) fixed Blog updating timestamp on edit (now always keeps creation-date) 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1997 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2006-04-04 15:10:59 +00:00
orbiter
22de954a57 added some log output to parser 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1996 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2006-04-04 15:01:21 +00:00
orbiter
83e0e765ec redesigned some parts of the html scanner & parser 
to better support image tags

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1995 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2006-04-04 14:36:01 +00:00
orbiter
ac114d69c0 tried to fix some problems with time-outs during search 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1994 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2006-04-03 23:26:08 +00:00
orbiter
e2e8d0c188 some kind of refactoring of yacysearch: 
made 'room' for new picture search result presentation

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1993 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2006-04-03 22:47:59 +00:00
orbiter
6b63e26cbb - removed search function from index.html/java, only imput left 
- added media fetcher/crawler class (not ready yet)


git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1992 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2006-04-03 15:36:53 +00:00
orbiter
bc3e80fe42 quickfix 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1990 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2006-04-02 23:03:17 +00:00
orbiter
d8d0ac29c3 added image-viewer servlet that can do: 
- each image that is requested is stored in the cache
- the image is taken from the cache if exists there
- the image can be scaled
The purpose of creation a scaled image is because of copyright problems
In a further stept the retrieval of not-shrinked images is restricted
to either access from localhost or with given authentication
This servlet can be used for image-preview purpose after an image search

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1989 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2006-04-02 22:59:53 +00:00
orbiter
ddc6394d9b fixed bug about auto-depth 0 
see http://www.yacy-forum.de/viewtopic.php?p=19751#19751

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1988 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2006-04-02 21:30:04 +00:00
orbiter
60351fa3f7 small fix to previous commit 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1987 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2006-04-02 20:52:34 +00:00
orbiter
a469874e3f added and fixed time-out behaviour during search 
see also: http://www.yacy-forum.de/viewtopic.php?p=19823#19823

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1986 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2006-04-02 20:40:07 +00:00
orbiter
1d0b0d6e2a synchronized local searched to prevent that several searches are performed at the same time 
see also: http://www.yacy-forum.de/viewtopic.php?p=19761#19761

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1985 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2006-04-02 18:51:18 +00:00
hermens
22b9d03bbf Correcting remaining time issue in getContainers 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1984 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2006-03-31 09:52:55 +00:00
orbiter
5defe97ccb fixed a problem / possible bug with ordering implementation (did not rotate always) 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1983 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2006-03-31 00:07:56 +00:00
orbiter
d58788b753 added some synchronisation 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1982 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2006-03-30 15:38:46 +00:00
orbiter
e566d1d8d6 some bugfixes regarding new crawling options 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1980 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2006-03-28 22:54:36 +00:00
orbiter
c7f1300300 -fixes for last commit 
-some more ranking attributes (comments only)

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1979 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2006-03-28 15:37:45 +00:00
orbiter
f2421f6a47 some small attribut changes regarding cache flush 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1974 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2006-03-27 23:14:04 +00:00
orbiter
7a650d0023 several bugfixes 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1971 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2006-03-27 16:45:29 +00:00
borg-0300
245cc34d51 small fix for last commit 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1969 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2006-03-26 18:21:41 +00:00
borg-0300
e2853f357d added more lowercase to url normal form generation 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1968 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2006-03-26 17:43:36 +00:00
orbiter
59d52fb4a9 fixed some problems with crawl profiles 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1967 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2006-03-26 14:52:01 +00:00
orbiter
4a5b5515b5 added a lowercase to url normal form generation 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1966 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2006-03-26 12:01:00 +00:00
allo
44d72f06c4 more Caching 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1965 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2006-03-24 15:08:27 +00:00
allo
918445a2f4 Bugfix for last commit. 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1964 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2006-03-24 14:34:27 +00:00
orbiter
708cc6c8d9 fixed some bugs for auto-filter and added monitor in profile list 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1959 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2006-03-24 00:38:40 +00:00
allo
c58789177f bookmarkCache 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1956 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2006-03-23 21:33:09 +00:00
rramthun
250864406f ... 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1955 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2006-03-23 20:24:53 +00:00
rramthun
42b0b10a95 -Adding Windows Media to types which are not sended compressed 
-Renaming writeandzip to writeandgzip to avoid confusion about type of compression
-Adding new startup message to windows script
-The usual language "enhancements" ;-)

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1953 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2006-03-23 20:12:23 +00:00
allo
2ed4fa96b7 tagCache 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1952 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2006-03-23 19:04:05 +00:00
allo
330eb9c74f bookmarkDB cleanup 
(preparation for tagCache)

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1951 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2006-03-23 17:56:32 +00:00
orbiter
e82899ba57 fixed missing urls map initializer 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1950 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2006-03-23 16:13:23 +00:00
orbiter
63f39ac7b5 added 3 new crawling steering options: 
- re-crawl by age of page (enter in minutes)
- auto-domain-filter
- maximum number of pages per domain
NOT YET TESTED!

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1949 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2006-03-23 16:05:16 +00:00
orbiter
1fc3b34be6 some pre-work (without function yet) to implement: 
- re-crawl (by age of last crawl)
- auto-crawl-filter by crawl depth (to be explained..)

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1948 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2006-03-22 15:28:17 +00:00
theli
c9e6b5e391 *) check size of indexing-queue and crawler pool before processing remote triggered crawl jobs 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1946 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2006-03-22 14:19:03 +00:00
orbiter
1509314ea6 set tighter control during DHT index and peer selection 
see http://www.yacy-forum.de/viewtopic.php?p=19329#19329

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1945 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2006-03-22 13:54:35 +00:00
hydrox
fcc0683200 *) undoing last commit 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1944 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2006-03-22 09:01:25 +00:00
hydrox
9411961eec *) another little fix for DHT-Transfer 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1943 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2006-03-22 08:49:39 +00:00
borg-0300
77f3237de3 adapted for isListed() 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1942 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2006-03-21 20:55:59 +00:00
hydrox
8b14a0c833 *) little fix for DHT-Transfer 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1941 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2006-03-21 10:35:57 +00:00
orbiter
1f4412a146 adopted isListed to discussed new behavior as discussed (url, getFile) 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1940 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2006-03-20 22:31:59 +00:00
orbiter
59fc55ea1e added checks to protect peers from wrong seeds 
see also: http://www.yacy-forum.de/viewtopic.php?p=19249#19249

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1939 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2006-03-20 20:31:51 +00:00
orbiter
063ef4660a bug? 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1936 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2006-03-19 22:06:15 +00:00
orbiter
82358677a9 added another shiftK2W to flushCacheSome 
this should fix the bug that the DHT cache is not flushed if there is no indexing

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1935 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2006-03-19 21:33:31 +00:00
orbiter
128e4ab199 - in serverSystem: maxPathLength is now a variable, not a method 
- upon startup the calculated maximum path length is shown

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1932 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2006-03-19 01:33:20 +00:00
orbiter
a37b09e303 implemented automatic adoption of chunk-read-ahead in kelondroTree to needed chunk size 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1931 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2006-03-19 01:27:33 +00:00
orbiter
30e3e3a0fd adopted MAXPATHLENGTH to host system capabilities 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1930 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2006-03-19 00:29:27 +00:00
borg-0300
d808765087 something Javadoc 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1929 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2006-03-18 21:47:04 +00:00
borg-0300
85bb8e32a1 Bugfix for last commit 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1928 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2006-03-18 19:05:56 +00:00
borg-0300
3fe402069f try to fix 
see: http://www.yacy-forum.de/viewtopic.php?p=19175#19175

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1927 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2006-03-18 18:51:52 +00:00
orbiter
f16f1f15cd bugfix for 100% CPU bug; thanks to Matthias for analysis 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1926 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2006-03-18 16:14:24 +00:00
borg-0300
254a13efd9 MAXPATHLENGTH used 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1925 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2006-03-18 15:20:50 +00:00
borg-0300
8865948e4e Cleanup; 
Methode replaceRegex added;
Constant MAXPATHLENGTH added;

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1923 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2006-03-18 13:34:32 +00:00
orbiter
6c70f4a0cf renamed wordHashes for a word hash set generation to wordHashSet 
This was done because the wordHashes iterator will get another integer
parameter and then conflicts with the wordHashes set generation

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1921 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2006-03-18 01:04:16 +00:00
orbiter
d5f8f40c31 removed correcting iterator 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1920 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2006-03-18 01:01:00 +00:00
orbiter
526407f32e adoptions, fixes for last commit 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1919 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2006-03-18 00:21:34 +00:00
orbiter
488a0ed580 replaced old keyIterator and rowIterator by buffered iterators 
that are synchronized with database access
Main change is done in kelondroTree, other classes are only adoptions

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1918 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2006-03-17 23:43:24 +00:00
hermens
4e9a8f41fd rwiDBCleaner + dbImporter: Iterate over small excerpts of 
word hashes instead of the whole DB especially while changing
the DB in the process.
see http://www.yacy-forum.de/viewtopic.php?p=19136#19136



git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1917 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2006-03-17 23:39:10 +00:00
hermens
474379ae63 remove TABs from plasmaDbImporter.java 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1916 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2006-03-17 21:52:36 +00:00
orbiter
dba02f399f starting of re-design of kelondroTree iterator 
- new access to iterator
- added many IOException handling in other Classes

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1914 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2006-03-17 20:52:43 +00:00
orbiter
f02b426073 made kelondroTree.nodeIterator private 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1910 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2006-03-17 18:10:48 +00:00
borg-0300
5f6fdf1786 Bugfix for getCachePath(URL url) 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1909 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2006-03-17 16:39:29 +00:00
orbiter
303b6463a8 added debug line to URL storage for testing 
see http://www.yacy-forum.de/viewtopic.php?p=19129#19129

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1908 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2006-03-17 16:30:31 +00:00
orbiter
431a4f3609 eliminated correcting iterator in kelondroTree 
VERY EXPERIMENTAL! NOT TESTED!

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1907 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2006-03-17 14:20:04 +00:00
orbiter
91dca2cd8d fixed a bug in last commit: LURL entries cannot be written, 
because a stored property was not set to false (but true)

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1906 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2006-03-17 13:07:34 +00:00
orbiter
3286b1f498 re-organisation of lurl-creation and -stacking 
this was necessary to prevent useless write to the database
in case of blacklist appearance of the url

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1905 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2006-03-17 10:16:07 +00:00
orbiter
9cca36a107 no more strict comparator checking in map exclude method if not needed 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1901 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2006-03-16 22:46:32 +00:00
orbiter
0b903c5317 removed usage of kelondroNaturalOrder from plasmaCondenser to experimental 
exclude cause of a 100% bug.
see http://www.yacy-forum.de/viewtopic.php?p=19076#19076

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1900 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2006-03-16 22:26:04 +00:00
orbiter
4239db0d1c fixed new ordering for backup iterator TreeSet 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1899 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2006-03-16 22:20:28 +00:00
borg-0300
399538b7de Bugfix: wrongly compared 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1898 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2006-03-16 20:57:06 +00:00
orbiter
33eba5ecb8 temporary disabling last change, does not work (cannot debug right now) 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1896 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2006-03-16 16:50:56 +00:00
orbiter
f0464042fc fix for latest iterator-replacement-fix: 
iterator generated TreeSet which did not resprect rotations
this has now be implemented using kelondroOrder Objects
and by adding this rotation-rules to the ording

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1895 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2006-03-16 16:44:29 +00:00
borg-0300
ec21c585cb try to fix path too long 
see http://www.yacy-forum.de/viewtopic.php?p=19079

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1893 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2006-03-15 20:16:15 +00:00
orbiter
a6a3f4b694 fix for svn 1888 
this is a redesign of the no-iterator solution

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1892 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2006-03-15 16:01:42 +00:00
hydrox
8da13088e9 *)removed multiple DHT_Distribution_Threads 
*)boosted DHT_Distribution sending chunk parallel to multiple peers

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1890 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2006-03-15 11:27:43 +00:00
orbiter
283a7181c6 try to fix new 100% cpu bug, possibly caused by iterator method 
see http://www.yacy-forum.de/viewtopic.php?p=18900#18900

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1888 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2006-03-14 23:22:49 +00:00
theli
34c075c1c7 testcommit with subversive 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1886 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2006-03-14 08:54:44 +00:00
orbiter
f588c0724f removed cache flush in case of DHT receive 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1885 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2006-03-13 23:23:45 +00:00
orbiter
e94b374d56 update to cache flush method 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1884 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2006-03-13 23:06:06 +00:00
orbiter
bcd99fe83e introduced a second RAM cache for DHT transfer 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1880 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2006-03-13 10:43:12 +00:00
hydrox
360a460da8 *)URL-Cleaner: moved logging-statement to correct position 
*)plasmaURLPattern: host is now added to the hashset in lowercase

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1879 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2006-03-12 18:08:48 +00:00
orbiter
02f9765013 quickfix for time problem during cache restore 
see http://www.yacy-forum.de/viewtopic.php?p=18810#18810

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1878 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2006-03-11 21:55:32 +00:00
orbiter
69d26045eb quick-fix for http://www.yacy-forum.de/viewtopic.php?p=18805#18805 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1876 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2006-03-10 20:45:14 +00:00
allo
b3df5ce2b8 another local iprange 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1875 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2006-03-10 18:34:53 +00:00
hermens
ad119f06af *) Don't overwrite new entries with older ones 
see: http://www.yacy-forum.de/viewtopic.php?t=2015



git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1874 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2006-03-10 16:28:01 +00:00
orbiter
be88687d8c fixed some problems with new cache flush karenz 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1873 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2006-03-10 13:57:30 +00:00
theli
d3da7c9a08 *) Adding support for robots Allow directive 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1872 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2006-03-09 14:03:54 +00:00
theli
734d18f283 *) more correct robots.txt validation 
- isDisallowed now uses getFile instead of getPath

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1870 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2006-03-09 13:04:57 +00:00
hydrox
f046e1814a *fix or last commit 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1869 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2006-03-09 12:35:50 +00:00
hydrox
c55c51e2a8 *)added keywords to IndexCleaner_p.java 
*)updated Logging

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1868 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2006-03-09 12:23:05 +00:00
theli
f0ad0d2b2b *) better robots.txt support 
- previously rules for all crawlers and special rules for yacy where 
     combined using AND. Now the general rule will be ignored if there is
     a special rule for yacy (according to rfc)

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1867 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2006-03-09 12:19:46 +00:00
orbiter
ddbeda738e added minimum age of word in cache to performance menu 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1866 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2006-03-09 11:31:17 +00:00
orbiter
f188611fc6 apply blacklist on rwis during dht receive 
very experimental!

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1865 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2006-03-09 10:46:02 +00:00
theli
915812f597 *) Undoing robots parser policy changes from svn rev. 1421 
- crawling is not allowed if server returned a 403 statuscode (according to rfc)

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1864 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2006-03-09 10:17:37 +00:00
orbiter
0ec28b8f8e added DBCleaner from Hydrox 
see http://www.yacy-forum.de/viewtopic.php?p=18093#18093
The servlet is now named IndexCleaner_p.
See http://localhost:8080/IndexCleaner_p.html
The Servlet was adopted to fit in the overall architecture

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1863 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2006-03-08 22:06:11 +00:00
theli
5ee0125046 *) adding possibility to configure the server port for seed uploading via scp. 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1861 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2006-03-08 16:34:05 +00:00
allo
2065138a41 reverted last Commit. 
@Orbiter: kannst du dich dann mal drum kümmern, wenn ich versuche die ganze Sache ans Laufen zu bringen, hast du jedesmal was dagegen. Dann mach du es bitte, du wirst ja wissen was du willst...


git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1860 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2006-03-08 13:35:38 +00:00
allo
7afa5c1b8e staticIP fix 
tried to solve http://www.yacy-forum.de/viewtopic.php?p=18663#18663
D 2006/03/08 07:08:20 YACY yacyClient.publishMySeed mySeed error - not proper: IP is not proper: -UNRESOLVED_PATTERN-


git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1859 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2006-03-08 12:23:26 +00:00
theli
fb4100d47b *) undoing last commit. 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1856 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2006-03-08 07:59:32 +00:00
theli
a84cc71218 *) removing getTotalRuntime 
- not needed anymore

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1855 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2006-03-08 07:37:21 +00:00
low012
1965a8cf70 *) bugfix for previous commit 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1853 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2006-03-08 00:38:39 +00:00