Marown/yacy_search_server
1
0
Fork 0
mirror of https://github.com/yacy/yacy_search_server.git synced 2024-09-21 00:00:13 +02:00
Code Issues Actions Packages Projects Releases Wiki Activity
4,226 Commits 1 Branch 20 Tags 836 MiB
677ee2ea04
Commit Graph

79 Commits

Author SHA1 Message Date
orbiter
4a80902081 - added ViewProfile as rdf in foaf syntax 
- added link to rdf and vCard version on html page
- can be seen on http://localhost:8080/ViewProfile.html?hash=localhash
- more generics

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4411 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-01-28 18:21:08 +00:00
orbiter
4ffbcd54a4 fix for http://forum.yacy-websuche.de/viewtopic.php?f=6&t=754 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4358 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-01-22 19:10:03 +00:00
hermens
4748d5c1ab Some enhancements to time management: 
- remove unnecessary generation of Calendar and Date objects
- synchronized SimpleDateFormat objects in blog-, message- and wikiBoard
- correct use of TimeZones and SimpleDateFormats



git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4288 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2007-12-20 17:11:35 +00:00
orbiter
daf0f74361 joined anomic.net.URL, plasmaURL and url hash computation: 
search profiling showed, that a major amount of time is wasted by computing url hashes. The computation does an intranet-check, which needs a DNS lookup. This caused that each urlhash computation needed 100-200 milliseconds, which caused remote searches to delay at least 1 second more that necessary. The solution to this problem is to attach a URL hash to the URL data structure, because that means that the url hash value can be filled after retrieval of the URL from the database. The redesign of the url/urlhash management caused a major redesign of many parts of the software. Since some parts had been decided to be given up they had been removed during this change to avoid unnecessary maintenance of unused code.

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4074 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2007-09-05 09:01:35 +00:00
orbiter
9ca46a8c69 indexing of local (intranet) urls enabled 
To do this, one must create a separate YaCy network that has a local URL domain
A description how to do this is here: http://www.yacy-websuche.de/wiki/index.php/De:Netzdefinition

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4001 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2007-07-24 00:46:17 +00:00
orbiter
ec817a2ff5 removed JAR command from ftpc - produces warnings and is not used by YaCy 
see http://forum.yacy-websuche.de/viewtopic.php?f=5&t=193&hilit=

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3997 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2007-07-22 20:08:09 +00:00
orbiter
40b0547611 - documentaton changes (removed old forum links) 
- different handling of link quotation
- different handling of link normalization
- enhanced html/unicode en/de-coding

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3993 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2007-07-19 15:32:10 +00:00
orbiter
36a37f758b fix for oom exception during release download 
see http://forum.yacy-websuche.de/viewtopic.php?f=6&t=101&hilit=

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3950 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2007-07-03 22:55:47 +00:00
orbiter
9bbd39b67c - removed unfinished auto-updater from roland and martin 
- added new download-option for releases on the status page
still mising:
- thomas-style restart for linux/mac
- untar/gunzip on shell basis
(comes next)

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3931 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2007-06-28 14:52:26 +00:00
karlchenofhell
03c6551b0c - fix for http://www.yacy-forum.de/viewtopic.php?t=3747 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3724 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2007-05-14 12:44:44 +00:00
orbiter
e602436fda fixed problem with cluster routing 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3684 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2007-05-07 20:48:24 +00:00
orbiter
871ee1ce0f one step closer to automatic updates: 
automatically aquire release information from download archives
web pages from latest.yacy-forum.net and yacy.net are retrieved, parsed,
links wihin are analysed, sorted and the most recent developer and main
releases are provided as direct download link on the status page, if it was
discovered that a more recent version than the current version is available.
This process is done only once during run-time of a peer, to protect our
download archives from DoS by YaCy peers.

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3606 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2007-04-27 09:23:44 +00:00
(no author)
4f4d3d71dd *) Faster appearance of ConfigBasic by bypassing UPNP-scan in case of existing external connects 
*) Marked two deprecated source-points
*) Added possibility to dump words from indexing to file. Should not affect performance in the current form.

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3592 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2007-04-24 16:33:31 +00:00
karlchenofhell
6fbe31425a - some code-cleanup (no more syntax-warnings here) 
- added deletion from loadedURLs of URLs to be blacklisted in IndexControl_p

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3404 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2007-02-26 12:56:50 +00:00
orbiter
f3f99b19c6 extended search statistics 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3249 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2007-01-19 01:45:29 +00:00
karlchenofhell
b3a2c79fec - fix for last fix 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3202 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2007-01-14 17:47:25 +00:00
karlchenofhell
4b23e79f51 - quick-fix for http://www.yacy-forum.de/viewtopic.php?t=3358 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3201 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2007-01-14 17:31:08 +00:00
karlchenofhell
be941c4475 - "javascript:"-URLs are recognized as well (as intended formerly I assume) 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3097 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2006-12-18 20:19:02 +00:00
karlchenofhell
a619ba3f49 - fix for String index out of range during URL parsing 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3096 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2006-12-18 19:32:01 +00:00
orbiter
d34f10c63d some tests with reverse dns lookup 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2954 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2006-11-12 00:28:10 +00:00
orbiter
bd4f43cd66 - fixed a null pointer exception bug 
- switched off more write caches
- re-enabled index-abstracts search

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2885 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2006-10-31 02:45:41 +00:00
theli
d38ef0493d *) be more tolerant against missing ports in url 
"http://yacy.net:/" is now interpreted as "http://yacy.net/"
   See: http://www.yacy-forum.de/viewtopic.php?p=27102

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2852 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2006-10-24 05:22:54 +00:00
theli
cfe54fedc7 *) Bugfix for resolveBackpath problem with tailing /.. 
*) Junit testclass for resolveBackpath testing 

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2850 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2006-10-24 05:07:34 +00:00
karlchenofhell
d13b381f83 - added mint-green skin 
- removed test-urls because of problems with text-encoding

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2832 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2006-10-21 10:24:32 +00:00
karlchenofhell
ce237aefad - assortment-sizes table from PerformanceQueues_p.html is not shown if not used 
- escape query- and fragment-part of an url as well
- new resolveBackpath for urls: http://www.yacy-forum.de/viewtopic.php?t=2679#24867

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2815 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2006-10-19 15:27:24 +00:00
karlchenofhell
b14a500b88 - removed debug output from PerformanceMemory_p 
- added URL escaping (tested, nevertheless watch out for possibly broken URLs)

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2797 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2006-10-18 14:51:37 +00:00
orbiter
bcf2b800b4 applied UTF-8 encoding parameter to yacy-internal protocol communication 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2694 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2006-10-02 13:35:38 +00:00
orbiter
5a40ea7866 refactoring of wget string list generation 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2692 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2006-10-02 09:59:20 +00:00
orbiter
df1629b05a - code cleanup 
- version 0.471
- moved surftipps to own web page


git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2676 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2006-09-29 22:27:20 +00:00
borg-0300
f18304ddd3 unused/not needed imports removes; 
properties added;

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2628 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2006-09-18 22:21:18 +00:00
orbiter
1e7fd48afd added size method to ftpc 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2508 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2006-09-07 12:21:41 +00:00
orbiter
d4c5e2af01 html-dirlist can now also be generated from existing connections 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2493 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2006-09-05 10:11:07 +00:00
theli
7930839594 *) URL.java: userinfo was not taken over when generating a new url from a base url and a rel. path 
*) CrawlWorker.java: using new dirhtml function of ftpc

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2492 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2006-09-05 05:17:57 +00:00
orbiter
17ba468165 added html dirlisting generation in ftpc.java: 
ftpc.dirhtml() generates a StringBuffer with a complete web page

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2491 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2006-09-05 00:11:59 +00:00
theli
ab5a9bee66 *) adding some copyright headers 
*) next step of restructuring for new crawlers
   - adding first testversion of ftp crawler class
   -- does not create a htCache entry yet

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2483 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2006-09-04 14:38:29 +00:00
theli
5847492537 *) next step of restructuring for new crawlers 
- IndexCreate_p.java: correcting problems with ftp urls
   - URL.java does not cutout the userinfo anymore 
    (needed to transport authentication info in ftp urls, e.g. ftp://username:pwd@ftp.irgendwas.de)
   - plasmaCrawlLoader.java: 
   -- hack to re enable https urls
   -- adding function getSupportedProtocols

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2482 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2006-09-04 13:17:11 +00:00
orbiter
6cce47e217 test of ftp-urls in URL class 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2481 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2006-09-04 13:10:40 +00:00
orbiter
f933f00f09 another patch to URL protocol handling for 'news', 'nntp' etc: 
reject it! (the java.net.URL class rejects them too)

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2432 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2006-08-21 01:04:04 +00:00
orbiter
4c6e00d80a more bugfixes for URL class, see: 
http://www.yacy-forum.de/viewtopic.php?p=24844#24844

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2431 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2006-08-21 00:23:39 +00:00
orbiter
b7dc251948 fixed bugs in url class: 
- correct backpath ('..') handling
- correct absolute path handling
- included https


git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2428 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2006-08-19 22:27:01 +00:00
orbiter
276225d79e fix for URL class 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2423 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2006-08-18 21:33:00 +00:00
orbiter
f43c90fa98 fixed handling of null referer in crawlOrder 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2384 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2006-08-11 21:46:34 +00:00
orbiter
abf22f6e60 removed url normalform computation from htmlFilterContentScraper. 
This method was implemented in de.anomic.net.URL


git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2377 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2006-08-11 15:09:22 +00:00
theli
0db237467f *) bugfix for URL generation from file 
see: http://www.yacy-forum.de/viewtopic.php?p=24116

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2326 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2006-07-25 16:18:45 +00:00
orbiter
e20ff77c10 another bugfix in new url class 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2318 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2006-07-22 11:37:22 +00:00
orbiter
685430a1b5 bugfix in new URL class, better loggin for domain extraction 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2317 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2006-07-22 11:33:01 +00:00
orbiter
79af283f6c better debugging in new URL class for wrong port numbers 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2315 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2006-07-22 10:21:24 +00:00
orbiter
4bd626572b added hashCode and compareTo to new URL class 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2301 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2006-07-18 12:00:54 +00:00
theli
a70cbd959b *) further improvements for the anomic.net.url class 
- relpath starting with javascript: are ignored now
   - bugfix for concatenation of relpath starting with # or ?
     in this case no slash should be added to the baseURL, otherwise
     we get URLs of the form http://test.de/index.html/?param=value

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2298 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2006-07-18 05:12:08 +00:00
theli
8a1f1d96b3 *) Bugfix for url concatenation. Relative urls with / or http:// at the beginning 
were not handled correctly on url concatenation via new URL(URL,relPath).
   See: http://www.yacy-forum.de/viewtopic.php?t=2623

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2297 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2006-07-18 04:48:18 +00:00