Marown/yacy_search_server
1
0
Fork 0
mirror of https://github.com/yacy/yacy_search_server.git synced 2024-09-19 00:01:41 +02:00
Code Issues Actions Packages Projects Releases Wiki Activity
883a96f21a
yacy_search_server/htroot/ConfigHeuristics_p.html

140 lines
9.4 KiB
HTML
Raw Normal View History

added a configuration page for search heuristics. currently you can switch on there: - a site-operation heuristic that loads all direct links from a portal page if the site-operator is used - a direct crawl for search results from scroogle for the given search terms The configuration page can be found directly beside the network configuration page git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6951 6c8d7289-2bf4-0310-a012-ef5d649a1542
2010-06-27 23:38:16 +02:00
<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Strict//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-strict.dtd">
<html xmlns="http://www.w3.org/1999/xhtml">
<head>
Added German translation for ConfigHeuristics_p.html to de.lng Fixed Network -> Heuristics title tag of the page git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6963 6c8d7289-2bf4-0310-a012-ef5d649a1542
2010-07-06 00:58:51 +02:00
<title>YaCy '#[clientname]#': Heuristics Configuration</title>
added a configuration page for search heuristics. currently you can switch on there: - a site-operation heuristic that loads all direct links from a portal page if the site-operator is used - a direct crawl for search results from scroogle for the given search terms The configuration page can be found directly beside the network configuration page git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6951 6c8d7289-2bf4-0310-a012-ef5d649a1542
2010-06-27 23:38:16 +02:00
#%env/templates/metas.template%#
</head>
<body id="ConfigNetwork">
#%env/templates/header.template%#
redesign of search page integration menu structure
2014-03-31 03:18:38 +02:00
#%env/templates/submenuRanking.template%#
added a configuration page for search heuristics. currently you can switch on there: - a site-operation heuristic that loads all direct links from a portal page if the site-operator is used - a direct crawl for search results from scroogle for the given search terms The configuration page can be found directly beside the network configuration page git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6951 6c8d7289-2bf4-0310-a012-ef5d649a1542
2010-06-27 23:38:16 +02:00
<h2>Heuristics Configuration</h2>
<p>
upd: to open more external links in new browser-tab
2013-12-26 01:16:53 +01:00
A <a href="http://en.wikipedia.org/wiki/Heuristic" target="_blank">heuristic</a> is an 'experience-based technique that help in problem solving, learning and discovery' (wikipedia). The search heuristics that can be switched on here are techniques that help the discovery of possible search results based on link guessing, in-search crawling and requests to other search engines.
added a configuration page for search heuristics. currently you can switch on there: - a site-operation heuristic that loads all direct links from a portal page if the site-operator is used - a direct crawl for search results from scroogle for the given search terms The configuration page can be found directly beside the network configuration page git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6951 6c8d7289-2bf4-0310-a012-ef5d649a1542
2010-06-27 23:38:16 +02:00
When a search heuristic is used, the resulting links are not used directly as search result but the loaded pages are indexed and stored like other content. This ensures that blacklists can be used and that the searched word actually appears on the page that was discovered by the heuristic.
</p>
ConfigHeuristics_p.html: XHTML 1.0 Strict Changes - added empty action tag to form - replaced name tags with id (name is not a valid tag in XHTML 1.0 Strict) - changed label for target (so now clicking on the labels also activates the checkboxes) de.lng: Test with Subversion properties #2 git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6982 6c8d7289-2bf4-0310-a012-ef5d649a1542
2010-07-22 00:40:34 +02:00
<form action=""><fieldset>
adjusted ConfigHeuristic and changed QueryGoal.getOriginalQueryString to .getQueryString - since specific heuristic Twitter & Blekko is not longer available or redundant with OpenSearchHeuristic, adjusted ConfigHeuristic to use OpensearchHeuristic settings only. For this the default OSD search target list is made available (copied) by default and the other configs are removed. - the return of QueryGoal.getOriginalQueryString includes the queryModifier, which are held separately in a modifier object, but in most (all) cases just the query term is expected, clarified and renamed it to QueryGoal.getQueryString which returns just the search term (if needed a .getOrigianlQueryString could be implemented in Queryparameters, adding the modifiers) - started to adjust internal html href references from absolute to relative (currently it is mixed). For future development we should prefer relative href targets (less trouble with context aware servlets)
2014-01-20 00:58:17 +01:00
The success of heuristics are marked with an image (<img width="16" height="9" src="env/grafics/heuristic_redundant.gif" title="heuristic:&lt;name&gt; (redundant)" style="width:16px; height:9px;" alt="heuristic:&lt;name&gt; (redundant)"/>/<img width="16" height="9" src="env/grafics/heuristic_new.gif" title="heuristic:&lt;name&gt; (new link)" style="width:16px; height:9px;" alt="heuristic:&lt;name&gt; (new link)"/>) below the favicon left from the search result entry:
added a configuration page for search heuristics. currently you can switch on there: - a site-operation heuristic that loads all direct links from a portal page if the site-operator is used - a direct crawl for search results from scroogle for the given search terms The configuration page can be found directly beside the network configuration page git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6951 6c8d7289-2bf4-0310-a012-ef5d649a1542
2010-06-27 23:38:16 +02:00
<dl>
<dt>
adjusted ConfigHeuristic and changed QueryGoal.getOriginalQueryString to .getQueryString - since specific heuristic Twitter & Blekko is not longer available or redundant with OpenSearchHeuristic, adjusted ConfigHeuristic to use OpensearchHeuristic settings only. For this the default OSD search target list is made available (copied) by default and the other configs are removed. - the return of QueryGoal.getOriginalQueryString includes the queryModifier, which are held separately in a modifier object, but in most (all) cases just the query term is expected, clarified and renamed it to QueryGoal.getQueryString which returns just the search term (if needed a .getOrigianlQueryString could be implemented in Queryparameters, adding the modifiers) - started to adjust internal html href references from absolute to relative (currently it is mixed). For future development we should prefer relative href targets (less trouble with context aware servlets)
2014-01-20 00:58:17 +01:00
<img width="16" height="9" src="env/grafics/heuristic_redundant.gif" title="heuristic:&lt;name&gt; (redundant)" style="width:16px; height:9px;" alt="heuristic:&lt;name&gt; (redundant)"/>
added a configuration page for search heuristics. currently you can switch on there: - a site-operation heuristic that loads all direct links from a portal page if the site-operator is used - a direct crawl for search results from scroogle for the given search terms The configuration page can be found directly beside the network configuration page git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6951 6c8d7289-2bf4-0310-a012-ef5d649a1542
2010-06-27 23:38:16 +02:00
</dt>
<dd>
The search result was discovered by a heuristic, but the link was already known by YaCy
</dd>
<dt>
adjusted ConfigHeuristic and changed QueryGoal.getOriginalQueryString to .getQueryString - since specific heuristic Twitter & Blekko is not longer available or redundant with OpenSearchHeuristic, adjusted ConfigHeuristic to use OpensearchHeuristic settings only. For this the default OSD search target list is made available (copied) by default and the other configs are removed. - the return of QueryGoal.getOriginalQueryString includes the queryModifier, which are held separately in a modifier object, but in most (all) cases just the query term is expected, clarified and renamed it to QueryGoal.getQueryString which returns just the search term (if needed a .getOrigianlQueryString could be implemented in Queryparameters, adding the modifiers) - started to adjust internal html href references from absolute to relative (currently it is mixed). For future development we should prefer relative href targets (less trouble with context aware servlets)
2014-01-20 00:58:17 +01:00
<img width="16" height="9" src="env/grafics/heuristic_new.gif" title="heuristic:&lt;name&gt; (new link)" style="width:16px; height:9px;" alt="heuristic:&lt;name&gt; (new link)"/>
added a configuration page for search heuristics. currently you can switch on there: - a site-operation heuristic that loads all direct links from a portal page if the site-operator is used - a direct crawl for search results from scroogle for the given search terms The configuration page can be found directly beside the network configuration page git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6951 6c8d7289-2bf4-0310-a012-ef5d649a1542
2010-06-27 23:38:16 +02:00
</dt>
<dd>
The search result was discovered by a heuristic, not previously known by YaCy
</dd>
</dl></fieldset></form>
ConfigHeuristics_p.html: XHTML 1.0 Strict Changes - added empty action tag to form - replaced name tags with id (name is not a valid tag in XHTML 1.0 Strict) - changed label for target (so now clicking on the labels also activates the checkboxes) de.lng: Test with Subversion properties #2 git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6982 6c8d7289-2bf4-0310-a012-ef5d649a1542
2010-07-22 00:40:34 +02:00
<form id="HeuristicFormSite" method="post" action="ConfigHeuristics_p.html" enctype="multipart/form-data" accept-charset="UTF-8">
added a configuration page for search heuristics. currently you can switch on there: - a site-operation heuristic that loads all direct links from a portal page if the site-operator is used - a direct crawl for search results from scroogle for the given search terms The configuration page can be found directly beside the network configuration page git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6951 6c8d7289-2bf4-0310-a012-ef5d649a1542
2010-06-27 23:38:16 +02:00
<fieldset>
<legend>
<input type="checkbox" name="site_check" id="site" onclick="window.location.href='ConfigHeuristics_p.html?#(site.checked)#site_on=::site_off=#(/site.checked)#'" value="site"#(site.checked)#:: checked="checked"#(/site.checked)# />
- added bootstrap css framework - adopted all YaCy administration pages to new framework - created new search page layout (working, but still work in progress) - old skin files are fully appliable! (and looking good) - target is a new style based on bootstrap examples, see /test.html - icons in YaCy may be replaced by glyphicons (to be done)
2014-03-18 13:42:31 +01:00
'site'-operator: instant shallow crawl
added a configuration page for search heuristics. currently you can switch on there: - a site-operation heuristic that loads all direct links from a portal page if the site-operator is used - a direct crawl for search results from scroogle for the given search terms The configuration page can be found directly beside the network configuration page git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6951 6c8d7289-2bf4-0310-a012-ef5d649a1542
2010-06-27 23:38:16 +02:00
</legend>
<p>
When a search is made using a 'site'-operator (like: 'download site:yacy.net') then the host of the site-operator is instantly crawled with a host-restricted depth-1 crawl.
That means: right after the search request the portal page of the host is loaded and every page that is linked on this page that points to a page on the same host.
Because this 'instant crawl' must obey the robots.txt and a minimum access time for two consecutive pages, this heuristic is rather slow, but may discover all wanted search results using a second search (after a small pause of some seconds).
</p>
</fieldset>
</form>
add search result heuristic. adding a crawl job with depth-1 for every displayed search result (crawling every external linked page of displayed search result pages)
2012-07-01 00:12:20 +02:00
<form id="HeuristicFormSearchResult" method="post" action="ConfigHeuristics_p.html" enctype="multipart/form-data" accept-charset="UTF-8">
<fieldset>
<table>
<tr>
<td>
<legend>
<input type="checkbox" name="searchresult_check" id="searchresult" onclick="window.location.href='ConfigHeuristics_p.html?#(searchresult.checked)#searchresult_on=::searchresult_off=#(/searchresult.checked)#'" value="searchresult"#(searchresult.checked)#:: checked="checked"#(/searchresult.checked)# />
- added bootstrap css framework - adopted all YaCy administration pages to new framework - created new search page layout (working, but still work in progress) - old skin files are fully appliable! (and looking good) - target is a new style based on bootstrap examples, see /test.html - icons in YaCy may be replaced by glyphicons (to be done)
2014-03-18 13:42:31 +01:00
search-result: shallow crawl on all displayed search results
add search result heuristic. adding a crawl job with depth-1 for every displayed search result (crawling every external linked page of displayed search result pages)
2012-07-01 00:12:20 +02:00
</legend>
</td>
<td>
<legend>
<input type="checkbox" name="searchresultglobal_check" id="searchresultglobal" onclick="window.location.href='ConfigHeuristics_p.html?#(searchresultglobal.checked)#searchresultglobal_on=::searchresultglobal_off=#(/searchresultglobal.checked)#'" value="siteresultglobal"#(searchresultglobal.checked)#:: checked="checked"#(/searchresultglobal.checked)# />
- added bootstrap css framework - adopted all YaCy administration pages to new framework - created new search page layout (working, but still work in progress) - old skin files are fully appliable! (and looking good) - target is a new style based on bootstrap examples, see /test.html - icons in YaCy may be replaced by glyphicons (to be done)
2014-03-18 13:42:31 +01:00
add as global crawl job
add search result heuristic. adding a crawl job with depth-1 for every displayed search result (crawling every external linked page of displayed search result pages)
2012-07-01 00:12:20 +02:00
</legend>
</td>
</tr>
</table>
<p>
When a search is made then all displayed result links are crawled with a depth-1 crawl.
This means: right after the search request every page is loaded and every page that is linked on this page.
If you check 'add as global crawl job' the pages to be crawled are added to the global crawl queue (remote peers can pickup pages to be crawled).
Default is to add the links to the local crawl queue (your peer crawls the linked pages).
</p>
</fieldset>
</form>
adjusted ConfigHeuristic and changed QueryGoal.getOriginalQueryString to .getQueryString - since specific heuristic Twitter & Blekko is not longer available or redundant with OpenSearchHeuristic, adjusted ConfigHeuristic to use OpensearchHeuristic settings only. For this the default OSD search target list is made available (copied) by default and the other configs are removed. - the return of QueryGoal.getOriginalQueryString includes the queryModifier, which are held separately in a modifier object, but in most (all) cases just the query term is expected, clarified and renamed it to QueryGoal.getQueryString which returns just the search term (if needed a .getOrigianlQueryString could be implemented in Queryparameters, adding the modifiers) - started to adjust internal html href references from absolute to relative (currently it is mixed). For future development we should prefer relative href targets (less trouble with context aware servlets)
2014-01-20 00:58:17 +01:00
Adding heuristic to get search results from configured systems which support opensearch specification - any system supporting opensearch specification can be configured - search query is only forwarded to remote system if not enough results available on local peer - discover function provided, checking the local Solr index for links to opensearchdescription files, to add to the config - sample config file with some general search engines with opensearch support
2012-12-29 08:24:48 +01:00
<fieldset>
<form id="HeuristicFormOpenSearch" method="post" action="ConfigHeuristics_p.html" enctype="multipart/form-data" accept-charset="UTF-8">
<legend>
<input type="checkbox" name="opensearch_check" id="opensearch" onclick="window.location.href='ConfigHeuristics_p.html?#(opensearch.checked)#opensearch_on=::opensearch_off=#(/opensearch.checked)#'" value="opensearch"#(opensearch.checked)#:: checked="checked"#(/opensearch.checked)# />
- added bootstrap css framework - adopted all YaCy administration pages to new framework - created new search page layout (working, but still work in progress) - old skin files are fully appliable! (and looking good) - target is a new style based on bootstrap examples, see /test.html - icons in YaCy may be replaced by glyphicons (to be done)
2014-03-18 13:42:31 +01:00
opensearch load external search result list from active systems below
Adding heuristic to get search results from configured systems which support opensearch specification - any system supporting opensearch specification can be configured - search query is only forwarded to remote system if not enough results available on local peer - discover function provided, checking the local Solr index for links to opensearchdescription files, to add to the config - sample config file with some general search engines with opensearch support
2012-12-29 08:24:48 +01:00
</legend>
<p>
update configheuristics_p.html text to state current opensearch heuristic function
2015-03-09 00:09:36 +01:00
When using this heuristic, then every new search request line is used for a call to listed opensearch systems.
Adding heuristic to get search results from configured systems which support opensearch specification - any system supporting opensearch specification can be configured - search query is only forwarded to remote system if not enough results available on local peer - discover function provided, checking the local Solr index for links to opensearchdescription files, to add to the config - sample config file with some general search engines with opensearch support
2012-12-29 08:24:48 +01:00
20 results are taken from remote system and loaded simultanously, parsed and indexed immediately.
To find out more about OpenSearch see <a href="http://www.opensearch.org" target="_blank">OpenSearch.org</a>
</p>
</form>
<form action="ConfigHeuristics_p.html" method="post" enctype="multipart/form-data" accept-charset="UTF-8">
<div>
<b>Available/Active Opensearch System</b>
replacing direct html table cellspacing with css set-up for cellspacing
2014-03-31 01:15:35 +02:00
<table class="sortable" border="0">
Adding heuristic to get search results from configured systems which support opensearch specification - any system supporting opensearch specification can be configured - search query is only forwarded to remote system if not enough results available on local peer - discover function provided, checking the local Solr index for links to opensearchdescription files, to add to the config - sample config file with some general search engines with opensearch support
2012-12-29 08:24:48 +01:00
<tr class="TableHeader" valign="bottom">
<td>Active</td>
<td>Title</td>
<td>Comment</td>
<td>Url <small>(format opensearch <a href="http://www.opensearch.org/Specifications/OpenSearch/1.1#OpenSearch_URL_template_syntax" target="_blank">Url template syntax</a>)</small></td>
<td>delete</td>
</tr>
#{osdcfg}#
<tr class="TableCell#(dark)#Light::Dark#(/dark)#">
<td align="center"><input type="checkbox" name="ossys_#[title]#" value="checked" #(checked)#::checked="checked"#(/checked)#/></td>
<td align="left"><b><a href="#[urlhostlink]#" target="_blank">#[title]#</b></a> </td>
<td align="left">#[comment]#</td>
<td align="left"><input type="text" name="ossys_url_#[title]#" value="#[url]#" size="70"/></td>
<td align="center"><input type="checkbox" name="ossys_del_#[title]#" value="checked" #(delchecked)#::checked="checked"#(/delchecked)#/></td>
</tr>
#{/osdcfg}#
<tr>
<td><small>new</small></td>
<td><input type="text" name="ossys_newtitle"/></td>
<td><input type="text" name="ossys_newcomment"/></td>
<td><input type="text" name="ossys_newurl" size="70"/></td>
<td><input type="submit" name="addnewosd" value="add"/></td>
</tr>
adjusted ConfigHeuristic and changed QueryGoal.getOriginalQueryString to .getQueryString - since specific heuristic Twitter & Blekko is not longer available or redundant with OpenSearchHeuristic, adjusted ConfigHeuristic to use OpensearchHeuristic settings only. For this the default OSD search target list is made available (copied) by default and the other configs are removed. - the return of QueryGoal.getOriginalQueryString includes the queryModifier, which are held separately in a modifier object, but in most (all) cases just the query term is expected, clarified and renamed it to QueryGoal.getQueryString which returns just the search term (if needed a .getOrigianlQueryString could be implemented in Queryparameters, adding the modifiers) - started to adjust internal html href references from absolute to relative (currently it is mixed). For future development we should prefer relative href targets (less trouble with context aware servlets)
2014-01-20 00:58:17 +01:00
<tr>
replaced pdblue-homebrew buttons with bootstrap standard buttons
2014-03-20 22:52:01 +01:00
<td colspan="3" style="padding-top:10px "><input type="submit" name="setopensearch" value="Save" class="btn btn-primary"/></td>
<td colspan="2" align="center" style="padding-top:10px "><input type="submit" name="resettodefaultosdlist" value="reset to default list" class="btn btn-primary"/></td>
adjusted ConfigHeuristic and changed QueryGoal.getOriginalQueryString to .getQueryString - since specific heuristic Twitter & Blekko is not longer available or redundant with OpenSearchHeuristic, adjusted ConfigHeuristic to use OpensearchHeuristic settings only. For this the default OSD search target list is made available (copied) by default and the other configs are removed. - the return of QueryGoal.getOriginalQueryString includes the queryModifier, which are held separately in a modifier object, but in most (all) cases just the query term is expected, clarified and renamed it to QueryGoal.getQueryString which returns just the search term (if needed a .getOrigianlQueryString could be implemented in Queryparameters, adding the modifiers) - started to adjust internal html href references from absolute to relative (currently it is mixed). For future development we should prefer relative href targets (less trouble with context aware servlets)
2014-01-20 00:58:17 +01:00
</tr>
Adding heuristic to get search results from configured systems which support opensearch specification - any system supporting opensearch specification can be configured - search query is only forwarded to remote system if not enough results available on local peer - discover function provided, checking the local Solr index for links to opensearchdescription files, to add to the config - sample config file with some general search engines with opensearch support
2012-12-29 08:24:48 +01:00
</table>
</div>
<div>
<span style="color:red">#[osderrmsg]#</span>
</div>
<br>
<div>
<div style="float:right">
replaced pdblue-homebrew buttons with bootstrap standard buttons
2014-03-20 22:52:01 +01:00
<input type="submit" name="discoverosd" id="discoverosd" value="discover from index" class="btn btn-primary" onclick="return confirm('start background task, depending on index size this may run a long time')"/>
Adding heuristic to get search results from configured systems which support opensearch specification - any system supporting opensearch specification can be configured - search query is only forwarded to remote system if not enough results available on local peer - discover function provided, checking the local Solr index for links to opensearchdescription files, to add to the config - sample config file with some general search engines with opensearch support
2012-12-29 08:24:48 +01:00
</div>
adjust Opensearch discover function to new webgraph Solr schema
2013-03-14 03:10:54 +01:00
With the button "discover from index" you can search within the metadata of your local index (Web Structure Index) to find systems which support the Opensearch specification.
Adding heuristic to get search results from configured systems which support opensearch specification - any system supporting opensearch specification can be configured - search query is only forwarded to remote system if not enough results available on local peer - discover function provided, checking the local Solr index for links to opensearchdescription files, to add to the config - sample config file with some general search engines with opensearch support
2012-12-29 08:24:48 +01:00
The task is started in the background. It may take some minutes before new entries appear (after refreshing the page).
Alternatively you may <a href="?copydefaultosdconfig=">copy &amp; paste a example config file</a> located in <i>defaults/heuristicopensearch.conf</i> to the DATA/SETTINGS directory.
adjust Opensearch discover function to new webgraph Solr schema
2013-03-14 03:10:54 +01:00
For the discover function the <i>web graph</i> option of the web structure index and the fields <i>target_rel_s, target_protocol_s, target_urlstub_s</i> have to be switched on in the <a href="IndexSchema_p.html?core=webgraph">webgraph Solr schema</a>.
add hint to Heuristics Config on "Greedy Learning Mode" in portal config, to point to a option to make this setting permanent.
2014-12-22 20:36:29 +01:00
#{osdsolrfieldswitch}#<input type="submit" name="switchsolrfieldson" value="switch Solr fields on" class="btn btn-primary btn-xs" onclick="return confirm('modify Solr Schema')"/>#{/osdsolrfieldswitch}#
Adding heuristic to get search results from configured systems which support opensearch specification - any system supporting opensearch specification can be configured - search query is only forwarded to remote system if not enough results available on local peer - discover function provided, checking the local Solr index for links to opensearchdescription files, to add to the config - sample config file with some general search engines with opensearch support
2012-12-29 08:24:48 +01:00
</div>
</form>
</fieldset>
added a configuration page for search heuristics. currently you can switch on there: - a site-operation heuristic that loads all direct links from a portal page if the site-operator is used - a direct crawl for search results from scroogle for the given search terms The configuration page can be found directly beside the network configuration page git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6951 6c8d7289-2bf4-0310-a012-ef5d649a1542
2010-06-27 23:38:16 +02:00
#%env/templates/footer.template%#
</body>
Scroogle is not comming back, remove dead code Conflicts: source/net/yacy/search/Switchboard.java
2012-06-10 23:38:41 +02:00
</html>
Reference in New Issue Copy Permalink