Commit Graph

6996 Commits

Author SHA1 Message Date
orbiter
b8aee6d402 performance hacks for better search performance
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7230 6c8d7289-2bf4-0310-a012-ef5d649a1542
2010-10-08 23:50:28 +00:00
orbiter
11bebe356b fixed crawl start: with SVN 7225 the name of the crawl start url was not given in input field and therefore all crawl starts had contained the empty string as crawl start url
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7229 6c8d7289-2bf4-0310-a012-ef5d649a1542
2010-10-08 22:02:24 +00:00
orbiter
2971c91988 fix for http://forum.yacy-websuche.de/viewtopic.php?p=20977#p20977
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7228 6c8d7289-2bf4-0310-a012-ef5d649a1542
2010-10-08 21:44:28 +00:00
orbiter
091dd3f6ec - enhanced intranet search speed
- enhanced intranet portscan speed (better time-out)

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7227 6c8d7289-2bf4-0310-a012-ef5d649a1542
2010-10-08 10:54:13 +00:00
low012
b9f405d1e8 *) added comments
*) more beautyful and easier to understand code (IMO)
*) added display= parameter to a lot of links in Wiki.html

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7226 6c8d7289-2bf4-0310-a012-ef5d649a1542
2010-10-08 00:32:50 +00:00
mikeworks
70576e88d2 de.lng: Added some more untranslated strings I found and uncommented old ones that were removed
terminal_p.html: Put back the old ID which was really easy to find
IndexCreate.js: Because XHTML 1.0 Strict does not allow name tags for some elements rewrote most element access functions to use getElementById
Table_API_p.html and all other html pages: Some XHTMl 1.0 Strict fixes, changed checkAll javascript, marked the first row with checkboxes as unsortable where applicable
Table_API_p.java and all other java pages: URLencoded lines with possible ampersands & -> & for validation XHTML 1.0 Strict sourcecode
--> All Index Create pages should validate now. Hope I did not break anything else (too much :-)


git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7225 6c8d7289-2bf4-0310-a012-ef5d649a1542
2010-10-06 00:00:23 +00:00
orbiter
efa59250f8 release 0.98 for SuMa-eV Demo tomorrow
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7224 6c8d7289-2bf4-0310-a012-ef5d649a1542
2010-10-05 17:52:54 +00:00
orbiter
6e6994e328 latest bugfixes to search and indexing function after test of demo presentation
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7223 6c8d7289-2bf4-0310-a012-ef5d649a1542
2010-10-05 17:49:53 +00:00
orbiter
c3bf17a3a1 fixed must-match filter for smb crawling
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7222 6c8d7289-2bf4-0310-a012-ef5d649a1542
2010-10-05 00:05:08 +00:00
orbiter
099def2a04 small changes in search widget appearance
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7221 6c8d7289-2bf4-0310-a012-ef5d649a1542
2010-10-04 23:43:33 +00:00
orbiter
50586a0dfd rename of widget to 'widget'
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7220 6c8d7289-2bf4-0310-a012-ef5d649a1542
2010-10-04 19:57:29 +00:00
apfelmaennchen
dffa142529 Fix for author navigator in yacyui-portalsearch.js
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7219 6c8d7289-2bf4-0310-a012-ef5d649a1542
2010-10-04 19:27:12 +00:00
orbiter
574346f8ce better must-match pattern for intranet file-crawls
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7218 6c8d7289-2bf4-0310-a012-ef5d649a1542
2010-10-04 12:55:39 +00:00
orbiter
aacf572a26 - enhancements for search speed
- bug fixes in many classes including basic data structure classes

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7217 6c8d7289-2bf4-0310-a012-ef5d649a1542
2010-10-04 11:54:48 +00:00
sixcooler
aa6075402a smal fix for crawling from 'sitelist' at changes from 7214
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7216 6c8d7289-2bf4-0310-a012-ef5d649a1542
2010-10-01 22:41:28 +00:00
sixcooler
61c82f3105 gzip-compresson @ transferRWI & transferURL back again
This reduce upload-volume to suit limited bandwidth of home-users like me :-)

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7215 6c8d7289-2bf4-0310-a012-ef5d649a1542
2010-10-01 00:42:43 +00:00
orbiter
2c549ae341 fixed a number of small bugs:
- better crawl star for files paths and smb paths
- added time-out wrapper for dns resolving and reverse resolving to prevent blockings
- fixed intranet scanner result list check boxes
- prevented htcache usage in case of file and smb crawling (not necessary, documents are locally available)
- fixed rss feed loader
- fixes sitemap loader which had not been restricted to single files (crawl-depth must be zero)
- clearing of crawl result lists when a network switch was done
- higher maximum file size for crawler

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7214 6c8d7289-2bf4-0310-a012-ef5d649a1542
2010-09-30 23:57:58 +00:00
orbiter
f6eebb6f99 replaced auto-dom filter with easy-to-understand Site Link-List crawler option
- nobody understand the auto-dom filter without a lenghtly introduction about the function of a crawler
- nobody ever used the auto-dom filter other than with a crawl depth of 1
- the auto-dom filter was buggy since the filter did not survive a restart and then a search index contained waste
- the function of the auto-dom filter was in fact to just load a link list from the given start url and then start separate crawls for all these urls restricted by their domain
- the new Site Link-List option shows the target urls in real-time during input of the start url (like the robots check) and gives a transparent feed-back what it does before it can be used
- the new option also fits into the easy site-crawl start menu

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7213 6c8d7289-2bf4-0310-a012-ef5d649a1542
2010-09-30 12:50:34 +00:00
mikeworks
63e387508c ConfigLanguage_p.java: Fixed the filename for the API call to ConfigLanguage_p.html - previously ConfigLanguage.html was recorded and the action could not be replayed with error 404 - Not found
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7212 6c8d7289-2bf4-0310-a012-ef5d649a1542
2010-09-30 03:16:17 +00:00
mikeworks
f468d377d7 Collage.html and Collage.java: Added <p> in body before images for XHTML 1.0 Transitional validationg and alt tag to images as well as closing tag <img (...) />
terminal_p.html: Set new link for starting a crawl to CrawlStartSite_p.html and replaced the old embed object of the Among.us Flash object by their new JS which takes care of adding the object correctly
de.lng: Moved the translations for the JS part from yacyinteractive.html to the yacyinteractive.js part
--> Terminal page is now valid XHTML 1.0 Transitional


git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7211 6c8d7289-2bf4-0310-a012-ef5d649a1542
2010-09-29 23:11:29 +00:00
orbiter
3057a0b939 - intranet scanner now produces urls with host names, not ips if possible
- CrawStartIntranet servlet shows IPs and host names

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7210 6c8d7289-2bf4-0310-a012-ef5d649a1542
2010-09-29 22:44:49 +00:00
orbiter
75964909aa added missing path to htroot (may only be necessary for cross-linking of servlet classes)
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7209 6c8d7289-2bf4-0310-a012-ef5d649a1542
2010-09-29 22:19:41 +00:00
mikeworks
421aa6a8bb ConfigLiveSearch.html: Fixed some HTML problems to validate at least XHTML 1.0 Transitional - strict is not possible because iframes are used. Replacing iframes with embedded object tag does not work in IE
ConfigPortal.html: Fixed some HTML problems to validate at least XHTML 1.0 Transitional - for strict the target attribute of the a link has to be removed
yacyinteractive.html: Moved all JS code to an external yacyinteractive.js file in JS folder
yacysearch.html: Removed embedded scripts from in between the body tags - now everything is loaded in the header
de.lng: Just in case JS files will be parsed at some point added translation for yacyinteractive.html result counter

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7208 6c8d7289-2bf4-0310-a012-ef5d649a1542
2010-09-29 20:06:39 +00:00
orbiter
c60aed4435 no caching in browser of dynamic web pages sent by YaCy http
this may prevent unnecessary IO caused by cache storage of the browser

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7207 6c8d7289-2bf4-0310-a012-ef5d649a1542
2010-09-29 19:56:42 +00:00
mikeworks
41a93ff565 de.lng: Small changes to the German translation and added missing part in Surftips.html and YaCy statistics link in left menu in header.template
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7206 6c8d7289-2bf4-0310-a012-ef5d649a1542
2010-09-29 06:08:19 +00:00
mikeworks
b7bb0cabaf Blacklist_p.html: Minor HTML and Javascript changes to get XHTML 1.0 Strict validation, lowercae onchange, id tags instead of name tags
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7205 6c8d7289-2bf4-0310-a012-ef5d649a1542
2010-09-29 05:41:32 +00:00
mikeworks
cd505d7e30 de.lng: German translation of the new Intranet Servlet introduced in SVN 7203 in CrawlStartIntranet_p.html
CrawlStartIntranet_p.html: New Intranet Crawl Start Servlet - minor HTML changes to get XHTML 1.0 Strict validation, remove (double) name tags, remove single ending </dt>

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7204 6c8d7289-2bf4-0310-a012-ef5d649a1542
2010-09-28 22:05:12 +00:00
orbiter
e63896f2a8 added an intranet scanner and a servlet which shows all intranet addresses and an option to start a site-crawl for all these addresses at once.
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7203 6c8d7289-2bf4-0310-a012-ef5d649a1542
2010-09-28 12:18:54 +00:00
orbiter
e54cb7fb0c more bugfixes (also for latest commit)
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7202 6c8d7289-2bf4-0310-a012-ef5d649a1542
2010-09-28 10:20:46 +00:00
orbiter
be6b48311c misc bugfixes
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7201 6c8d7289-2bf4-0310-a012-ef5d649a1542
2010-09-28 10:00:33 +00:00
suessthomas
44874f2cb9 Added "encoding =" UTF-8 "in the RSS files
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7200 6c8d7289-2bf4-0310-a012-ef5d649a1542
2010-09-27 20:36:49 +00:00
orbiter
d2fd93135c - moved yacybot user agent string definition to MultiProtocolURI since there are basic access mechanisms where the bot string is needed
- migrated the 'yacy' user agent to 'yacybot' in many client methods since the 'yacy' user agent is only used for the proxy

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7199 6c8d7289-2bf4-0310-a012-ef5d649a1542
2010-09-27 14:54:32 +00:00
orbiter
fc3ee9d8a1 changed paths for new libs also in Info.plist
when changing paths in build.xml, please also update the paths in /addon/YaCy.app/Contents/Info.plist

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7198 6c8d7289-2bf4-0310-a012-ef5d649a1542
2010-09-27 14:49:51 +00:00
sixcooler
f4357dff03 bump to httpclient-4.0.3 which fixes a number of bugs
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7197 6c8d7289-2bf4-0310-a012-ef5d649a1542
2010-09-27 13:24:40 +00:00
mikeworks
ad7efe6016 rssTerminal.html: Fixing the 'null' is null or not an object in rss2.js when viewing the YaCy default Status page http://localhost:8080/Status.html with Internet Explorer
feed.xml: copy of feed.rss that helps Internet Explorer also read the Feed - workaround for the fix above
Problem is described in the forums and should be fixed better ;-(http://forum.yacy-websuche.de/viewtopic.php?f=6&t=2766&p=20702)

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7196 6c8d7289-2bf4-0310-a012-ef5d649a1542
2010-09-26 22:55:52 +00:00
mikeworks
190de644dd de.lng: Added German translations for some missing table content on Network view
WatchWebStructure_p.html: Added JS verification of RGB color codes (currently only RGB value is checked but this could be enhanced to also check for websafe colors)

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7195 6c8d7289-2bf4-0310-a012-ef5d649a1542
2010-09-26 22:28:32 +00:00
orbiter
d5dc88a351 shop cleanup button only if servlet was called without post/put arguments.
This should avoid confusion after a search for a word where it is possible to delete the word. If a delete button is shown to delete the word, then there should not be a button available to delete the whole index to avoide a wrong usage when a user searches only for a word to delete it.

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7194 6c8d7289-2bf4-0310-a012-ef5d649a1542
2010-09-26 21:11:08 +00:00
low012
afa708d552 *) added <s>...</s> tag to WikiCode -> works just as the HTML equivalent
*) code changes (PMD) without functional changes

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7193 6c8d7289-2bf4-0310-a012-ef5d649a1542
2010-09-26 12:57:07 +00:00
orbiter
a83186ac7d fix for bug in cytrails
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7192 6c8d7289-2bf4-0310-a012-ef5d649a1542
2010-09-26 10:32:40 +00:00
mikeworks
b019426811 de.lng: Added German translations for new Index Creation pages RSS Feeds and adapted text in Tables_p.html and CrawlStartExpert_p.html to match some typos, also changed one name tag to id to conform with XHTML 1.0 Strict
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7191 6c8d7289-2bf4-0310-a012-ef5d649a1542
2010-09-26 01:39:51 +00:00
orbiter
48c0d508ac fixes for crawling of smb links (file length not always available)
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7190 6c8d7289-2bf4-0310-a012-ef5d649a1542
2010-09-25 22:32:26 +00:00
mikeworks
965a40b623 de.lng: Added some German translation for Admin Console -> Advanced Settings and started to add some translations for new Index Creation pages
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7189 6c8d7289-2bf4-0310-a012-ef5d649a1542
2010-09-25 21:53:13 +00:00
orbiter
0bc6284e27 - added bugfix for access tracker in case of concurrency conflicts
- added missing entry for new icu4j path in Mac App

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7188 6c8d7289-2bf4-0310-a012-ef5d649a1542
2010-09-25 21:10:50 +00:00
f1ori
6295cac35a set reasonable mime-type for icu4j-core.jar
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7187 6c8d7289-2bf4-0310-a012-ef5d649a1542
2010-09-25 20:04:34 +00:00
f1ori
e670e1ef8e add charset auto-detection for htmlParser
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7186 6c8d7289-2bf4-0310-a012-ef5d649a1542
2010-09-25 18:02:21 +00:00
f1ori
ddcd5ae78c fix for http://forum.yacy-websuche.de/viewtopic.php?f=5&t=2989
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7185 6c8d7289-2bf4-0310-a012-ef5d649a1542
2010-09-25 15:07:25 +00:00
f1ori
8fe1102452 fix http://forum.yacy-websuche.de/viewtopic.php?p=20889#p18426
reuse code from htmlParser

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7184 6c8d7289-2bf4-0310-a012-ef5d649a1542
2010-09-24 13:14:20 +00:00
orbiter
daeea96aea renamed servlet CrawlStart_p.html to CrawlStartSite_p.html to circumvent problem with translation which still showed old expert crawl start page
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7183 6c8d7289-2bf4-0310-a012-ef5d649a1542
2010-09-22 21:46:31 +00:00
orbiter
10a9cb1971 simplified snippet computation process and separated the algorithm into two classes
also enhances selection criteria for best snippet line computation

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7182 6c8d7289-2bf4-0310-a012-ef5d649a1542
2010-09-22 20:50:02 +00:00
lotus
4450c240b7 npe fix http://forum.yacy-websuche.de/viewtopic.php?f=6&t=2982
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7181 6c8d7289-2bf4-0310-a012-ef5d649a1542
2010-09-22 20:24:07 +00:00