Marown/yacy_search_server
1
0
Fork 0
mirror of https://github.com/yacy/yacy_search_server.git synced 2024-09-21 00:00:13 +02:00
Code Issues Actions Packages Projects Releases Wiki Activity
3,210 Commits 1 Branch 20 Tags 836 MiB
6fbe31425a
Commit Graph

3210 Commits

This Branch
This Branch
All Branches
Author SHA1 Message Date
karlchenofhell
6fbe31425a - some code-cleanup (no more syntax-warnings here) 
- added deletion from loadedURLs of URLs to be blacklisted in IndexControl_p

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3404 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2007-02-26 12:56:50 +00:00
orbiter
32867580ee update to kelondroRecords needed fo last commit 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3403 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2007-02-26 11:55:36 +00:00
orbiter
e3480d4ad3 fix for warning in crawl balancer 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3402 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2007-02-26 11:54:43 +00:00
daburna
ed021a3f70 bugfix, see http://www.yacy-forum.de/viewtopic.php?t=3573 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3401 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2007-02-26 10:05:28 +00:00
karlchenofhell
31ad42535a - added buttons to add complete domain or single URL to blacklist to IndexControl_p 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3400 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2007-02-25 23:14:45 +00:00
orbiter
8668ac5d91 preparations for collection index cache flush optimization 
(hand-over commit, no functional change to current code)

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3399 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2007-02-25 21:06:26 +00:00
allo
42e9747650 fixed /path/forwarding. uncomment, if you want to use it. 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3398 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2007-02-25 20:07:35 +00:00
karlchenofhell
e0decf4653 - added support for changing invalid entries in blacklist cleaner 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3397 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2007-02-25 19:36:05 +00:00
karlchenofhell
c58ef48e1c - increased size of subject text-field 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3396 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2007-02-25 18:48:25 +00:00
karlchenofhell
1d31ebbeec - added experimental PHP script which redirects from a vhost to a peer, using a public seed-file 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3395 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2007-02-25 15:18:22 +00:00
auron_x
9cbf94222f *) added seedurl to network.xml as requested by lulabad 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3394 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2007-02-25 10:24:42 +00:00
karlchenofhell
39a2000d8b - added support for [[Bookmark:$bookmarkTag|description]]-link-listings (requested by theli) to wiki-parser 
- added support for <pre>-tags to wiki-parser

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3393 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2007-02-24 21:26:48 +00:00
karlchenofhell
619653c054 - fix for last commit 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3392 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2007-02-24 15:40:56 +00:00
karlchenofhell
26f5757b40 - added support for multiple paths per domain to default-blacklist 
warning: an interface-change had been neccessary:
- remove(String, String) has been renamed to removeAll(String, String), because it removes all path-entries for the specified host
- remove(String, String, String) has been added to delete only a path-entry
- geBlacklistType(String) has been renamed to getBlacklistType(String)

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3391 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2007-02-24 13:56:32 +00:00
karlchenofhell
3d6ab19f7e - remove double entries in blacklist as well 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3390 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2007-02-23 18:27:56 +00:00
karlchenofhell
bf7a69197d - fix for possible NPE in queues_p 
- WatchCrawler_p:
  - display crawler traffic
  - pause/resume local- and global crawler


git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3389 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2007-02-22 22:26:11 +00:00
allo
9702d3abba further supertemplate test 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3388 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2007-02-22 11:01:16 +00:00
karlchenofhell
a5a36d9252 - hopefully last fix fo 1.5 methods (sorry for that, eclipse isn't that helpful in identifying those methods) 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3387 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2007-02-22 08:04:09 +00:00
karlchenofhell
e97b6f0458 - we still use Java 1.4 ... 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3386 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2007-02-21 22:43:31 +00:00
karlchenofhell
0c7b8cf632 - added first version of new wiki-parser 
- added blacklist support to manual URLFetcher stack fill
- fix for NPE: http://www.yacy-forum.de/viewtopic.php?t=3559

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3385 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2007-02-21 22:31:36 +00:00
orbiter
f7803a6ce4 enhanced crawl balancer 
- new domains now get a chance to get crawled early
- less IO operations
- new balancing method
- better dump order at shutdown time
- bugfixes regarding not found url hashes (no more superfluous cache kill)
- domain access time is now shared over all balancer stacks
- viewing the stack does no more disturbish the balancing algorithm that much
- intelligent selection of best next domain using domain access times
- extra double-check (to double-check the double-check)

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3384 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2007-02-21 16:23:31 +00:00
low012
801eea8849 *) Fixed bug where pairReplace() got caught in infinite recursion. (http://www.yacy-forum.de/viewtopic.php?t=3466) 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3383 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2007-02-20 22:07:59 +00:00
theli
c8862e47fb *) adding mimetype for svg 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3382 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2007-02-20 15:25:46 +00:00
orbiter
39b0658839 Redesign of Webinterface menu structure 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3381 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2007-02-20 14:35:29 +00:00
orbiter
c3e8c23f5d fix for 'CANNOT FETCH ENTRY: hash is null' bug 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3380 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2007-02-20 13:53:21 +00:00
orbiter
badab8d924 fixed some more bugs in new db handling 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3379 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2007-02-20 12:29:12 +00:00
orbiter
e72d253577 fixed problem with initial cache load 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3378 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2007-02-20 11:20:48 +00:00
orbiter
2d8e472cfd emergeny bugfix for last commit 
(kelondroTree should work again)
the cache prefill is broken and will be fixed later

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3377 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2007-02-20 10:25:17 +00:00
orbiter
868aaabf88 documentation update 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3376 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2007-02-20 08:37:09 +00:00
orbiter
dc0c06e43d PLEASE MAKE A BACK-UP OF YOUR COMPLETE DATA DIRECTORY BEFORE USING THIS 
redesign for better IO performance
enhanced database seek-time by avoiding write operations at distant
positions of a database file. until now, a USEDC counter was written
at the head-section of a kelondroRecords database file (which is the
basic data structure of all kelondro database files) to store the
actual number of records that are contained in the database. Now, this
value is computed from the database file size. This is either done
only once at start-time, or continuously when run in asserts enabled.
The counter is then updated only in RAM, and written at close of the
file. If the close fails, the correct number can be computed from the
file size, and if this is not equal to the stored number it is a strong
evidence that YaCY was not shut down properly.
To preserve consistency, the complete storage-routine had to be re-written.
Another change enhances read of nodes in some cases, where the data-tail
can be read together with the data-head. This saves another IO lookup during
each DB node fetch.
Includes also many small bugfixes.
IF ANYTHING GOES WRONG, ALL YOUR DATA IS LOST: PLEASE MAKE A BACK-UP

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3375 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2007-02-20 08:35:51 +00:00
hydrox
5af76fccd7 *) peer-search on Network.html now is case-insensitive 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3374 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2007-02-19 13:00:41 +00:00
karlchenofhell
c016fcb10f - added streaming-support to CrawlURLFetchStack_p servlet 
- bug for NPE in list.java
- use more constants

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3373 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2007-02-19 12:47:46 +00:00
karlchenofhell
65af9d3215 - continue shifting even in the case the stacked URL could not be found 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3372 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2007-02-18 01:50:41 +00:00
rramthun
fdd1180ac5 Adding two icon files, both containing different sizes from 16x16 to 128x128 pixels in one file. 
The .icns is for Macintosh

Both made by Philipp Redeker

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3371 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2007-02-17 22:21:11 +00:00
karlchenofhell
d114a0136e - crawl profile: don't add null-values 
- added some settings and statistics for url-fetcher 'server'-mode
- added own stack for fetchable URLs
- added possibility to fill stack via shift from peer's queues, via POST (addurls=$count and url$num=$url) or via file-upload
- added "htroot" to classpath of linux start-script

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3370 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2007-02-17 19:16:53 +00:00
karlchenofhell
a46dc43f45 - added lock symbol for restart- and stutdown-buttons on Status-page (see http://www.yacy-forum.de/viewtopic.php?p=31444#31444) 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3369 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2007-02-16 00:25:45 +00:00
karlchenofhell
b2a9d32f29 why do I always forget some lines? sorry... 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3368 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2007-02-14 15:11:03 +00:00
karlchenofhell
e6ddf135bb - enabled fetching new crawls via /yacy/list.html?list=queueUrls for testing purposes 
- sent URLs are taken off the limit-stack (of the global crawl trigger) (may be moved somewhere else in future versions)
- added option to set the requested chunk-size

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3367 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2007-02-14 14:50:55 +00:00
karlchenofhell
67d96249b4 - fix for last commit 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3366 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2007-02-13 21:17:43 +00:00
karlchenofhell
c5a2ba3a23 - prepared URL fetch from other peers 
- more feedback for user

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3365 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2007-02-13 20:18:12 +00:00
daburna
661a7bb702 *updated translation for 
-network
-wiki

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3364 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2007-02-12 20:53:47 +00:00
auron_x
5ba531a722 *) higher precision for QPH also on status-page 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3363 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2007-02-11 09:33:39 +00:00
karlchenofhell
4e5eda6ef9 huch... 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3362 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2007-02-10 20:25:45 +00:00
karlchenofhell
50b59e312f - added experimental CrawlURLFetch_p-Servlet to fetch new URLs from a specified location (\n-seperated list). Requested by Theli. 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3361 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2007-02-10 20:20:00 +00:00
karlchenofhell
6c6375577e - fix for http://www.yacy-forum.de/viewtopic.php?t=3523 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3360 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2007-02-10 11:50:18 +00:00
karlchenofhell
ea20d8d7c5 - return to edited wiki-page after submit 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3359 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2007-02-09 19:47:21 +00:00
theli
e1edb23689 *) Bugfix for IllegalMonitorStateException 
See: http://www.yacy-forum.de/viewtopic.php?t=3522

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3358 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2007-02-09 19:32:49 +00:00
orbiter
bf69a721cb more protection against mis-use of YaCyHop interface: 
- target must not be at port 80
- target access not more than every 3 seconds
- requester may not access more than every 10 seconds

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3357 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2007-02-09 15:25:10 +00:00
orbiter
a15963ff98 better balancing: if element from top would force a busy waiting, 
an element from the bottom of the stack is used instead.

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3356 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2007-02-09 10:32:58 +00:00
orbiter
dda24fcb85 ups 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3355 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2007-02-09 09:55:21 +00:00