Marown/yacy_search_server
1
0
Fork 0
mirror of https://github.com/yacy/yacy_search_server.git synced 2024-09-19 00:01:41 +02:00
Code Issues Actions Packages Projects Releases Wiki Activity
275cff0cb7
yacy_search_server/htroot/IndexImportMediawiki_p.html

108 lines
5.7 KiB
HTML
Raw Normal View History

Used a text input for wiki dump import file selection. Using an HTML "file" input was confusing (as reported by promocore on YaCy forum : http://forum.yacy-websuche.de/viewtopic.php?f=5&t=5965) , and it only worked with MS IE/Edge on a local YaCy peer : - for security reasons some current major browsers such as Firefox or Chrome do not allow to send full file path information when using a file form input - the local file system selection popup doesn't make sense when you want to import a dump on a remote YaCy server
2017-04-11 07:34:17 +02:00
<!DOCTYPE html>
<html lang="en">
added a submenu to index administration to import a wikimedia dump (i.e. a dump from wikipedia) into the YaCy index: see http://localhost:8080/IndexImportWikimedia_p.html git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5930 6c8d7289-2bf4-0310-a012-ef5d649a1542
2009-05-08 09:54:10 +02:00
<head>
*) Replaced occurrences of "Wikimedia" with "MediaWiki" where applicable. (Thanks to the folks of 0x20.be for pointing this out.) *) Added description of where to place MediaWiki dump for import. git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7905 6c8d7289-2bf4-0310-a012-ef5d649a1542
2011-08-28 02:16:36 +02:00
<title>YaCy '#[clientname]#': MediaWiki Dump Import</title>
fix IndexImportMediawiki_p servlet's refresh header add url parameter to make sure no parameter are included in refresh url which could cause unwanted restart of import job see http://mantis.tokeek.de/view.php?id=591 comments
2015-10-25 05:41:25 +01:00
#%env/templates/metas.template%#
Improved MediaWiki dump import monitoring. When import thread is terminated : - now stop refreshing and stay on the monitoring page to give user a feedback after a long running import - added link to the next monitoring step : results from surrogates reader - added link to new import On the new import page, added a link on the eventual last import report.
2017-05-02 09:38:45 +02:00
#(refresh)#::<meta http-equiv="REFRESH" content="10;url=IndexImportMediawiki_p.html?report=" />
fix IndexImportMediawiki_p servlet's refresh header add url parameter to make sure no parameter are included in refresh url which could cause unwanted restart of import job see http://mantis.tokeek.de/view.php?id=591 comments
2015-10-25 05:41:25 +01:00
<!-- the url= removes http get parameters on refresh, preventing restart of import -->
Improved MediaWiki dump import monitoring. When import thread is terminated : - now stop refreshing and stay on the monitoring page to give user a feedback after a long running import - added link to the next monitoring step : results from surrogates reader - added link to new import On the new import page, added a link on the eventual last import report.
2017-05-02 09:38:45 +02:00
#(/refresh)#
added a submenu to index administration to import a wikimedia dump (i.e. a dump from wikipedia) into the YaCy index: see http://localhost:8080/IndexImportWikimedia_p.html git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5930 6c8d7289-2bf4-0310-a012-ef5d649a1542
2009-05-08 09:54:10 +02:00
</head>
*) Replaced occurrences of "Wikimedia" with "MediaWiki" where applicable. (Thanks to the folks of 0x20.be for pointing this out.) *) Added description of where to place MediaWiki dump for import. git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7905 6c8d7289-2bf4-0310-a012-ef5d649a1542
2011-08-28 02:16:36 +02:00
<body id="IndexImportMediawiki">
added a submenu to index administration to import a wikimedia dump (i.e. a dump from wikipedia) into the YaCy index: see http://localhost:8080/IndexImportWikimedia_p.html git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5930 6c8d7289-2bf4-0310-a012-ef5d649a1542
2009-05-08 09:54:10 +02:00
#%env/templates/header.template%#
Complete redesign of administration top-level menu. This follows two principles: - provide an easy tutorial-like "what should I do first" menu - provide all elements which are subject to most first questions to YaCy exibition people on top level: Resource limitation, Parser and Ranking settings I apologize to everyone who are used to the old style and need to find the menu items (again) after this change. I hope that this will make the interface more usable for new users who see a web indexer/crawler the first time.
2014-03-31 15:47:58 +02:00
#%env/templates/submenuIndexImport.template%#
*) Replaced occurrences of "Wikimedia" with "MediaWiki" where applicable. (Thanks to the folks of 0x20.be for pointing this out.) *) Added description of where to place MediaWiki dump for import. git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7905 6c8d7289-2bf4-0310-a012-ef5d649a1542
2011-08-28 02:16:36 +02:00
<h2>MediaWiki Dump Import</h2>
added a submenu to index administration to import a wikimedia dump (i.e. a dump from wikipedia) into the YaCy index: see http://localhost:8080/IndexImportWikimedia_p.html git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5930 6c8d7289-2bf4-0310-a012-ef5d649a1542
2009-05-08 09:54:10 +02:00
#(import)#
Improved MediaWiki dump import monitoring. When import thread is terminated : - now stop refreshing and stay on the monitoring page to give user a feedback after a long running import - added link to the next monitoring step : results from surrogates reader - added link to new import On the new import page, added a link on the eventual last import report.
2017-05-02 09:38:45 +02:00
<p>#(prevReport)#::<a href="IndexImportMediawiki_p.html?report=">Check</a> the last import report.#(/prevReport)#</p>
Improved error reports on various wiki dump prerequisites failure cases. Also added some JavaDoc.
2017-04-11 08:21:34 +02:00
<p>#(status)#<div class="alert alert-info" role="alert">No import thread is running, you can start a new thread here</div>
Extended Mediawiki dump import to remote URLs. When using a public HTTP URL in /IndexImportMediawiki_p.html, the remote file now is directly streamed and processed, allowing import of several GB dumps even with a low memory remote peer, and without need to manually download the dump file first.
2017-04-14 14:32:44 +02:00
::<div class="alert alert-danger" role="alert">Error : dump <abbr title="Uniform Resource Locator">URL</abbr> is malformed.</div>
Improved error reports on various wiki dump prerequisites failure cases. Also added some JavaDoc.
2017-04-11 08:21:34 +02:00
::<div class="alert alert-danger" role="alert">Error : file not found "#[sourceFile]#"</div>
::<div class="alert alert-danger" role="alert">Error : can not read file "#[sourceFile]#"</div>
::<div class="alert alert-danger" role="alert">Error : you selected a directory ("#[sourceFile]#")</div>
Added MediaWiki dump import scheduling feature. Checking the last modified date by default to prevent unnecessary long running operations.
2017-05-03 18:53:01 +02:00
::<div class="alert alert-danger" role="alert">Error : dump file ("#[sourceFile]#") was not modified since last import (#[lastImportDate]#).</div>
Improved error reports on various wiki dump prerequisites failure cases. Also added some JavaDoc.
2017-04-11 08:21:34 +02:00
#(/status)#</p>
Used a text input for wiki dump import file selection. Using an HTML "file" input was confusing (as reported by promocore on YaCy forum : http://forum.yacy-websuche.de/viewtopic.php?f=5&t=5965) , and it only worked with MS IE/Edge on a local YaCy peer : - for security reasons some current major browsers such as Firefox or Chrome do not allow to send full file path information when using a file form input - the local file system selection popup doesn't make sense when you want to import a dump on a remote YaCy server
2017-04-11 07:34:17 +02:00
<form action="IndexImportMediawiki_p.html" method="post" accept-charset="UTF-8" class="form-horizontal">
Extended Mediawiki dump import to remote URLs. When using a public HTTP URL in /IndexImportMediawiki_p.html, the remote file now is directly streamed and processed, allowing import of several GB dumps even with a low memory remote peer, and without need to manually download the dump file first.
2017-04-14 14:32:44 +02:00
<input type="hidden" name="transactionToken" value="#[transactionToken]#"/>
added a submenu to index administration to import a wikimedia dump (i.e. a dump from wikipedia) into the YaCy index: see http://localhost:8080/IndexImportWikimedia_p.html git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5930 6c8d7289-2bf4-0310-a012-ef5d649a1542
2009-05-08 09:54:10 +02:00
<fieldset>
Extended Mediawiki dump import to remote URLs. When using a public HTTP URL in /IndexImportMediawiki_p.html, the remote file now is directly streamed and processed, allowing import of several GB dumps even with a low memory remote peer, and without need to manually download the dump file first.
2017-04-14 14:32:44 +02:00
<legend>MediaWiki Dump File Selection</legend>
Used a text input for wiki dump import file selection. Using an HTML "file" input was confusing (as reported by promocore on YaCy forum : http://forum.yacy-websuche.de/viewtopic.php?f=5&t=5965) , and it only worked with MS IE/Edge on a local YaCy peer : - for security reasons some current major browsers such as Firefox or Chrome do not allow to send full file path information when using a file form input - the local file system selection popup doesn't make sense when you want to import a dump on a remote YaCy server
2017-04-11 07:34:17 +02:00
<p>
Added a link to MediaWiki dumps summary in import page for convenience
2018-08-08 08:11:02 +02:00
You can import <a href="https://dumps.wikimedia.org/backup-index-bydb.html" target="_blank">MediaWiki dumps</a> here. An example is the file
Extended Mediawiki dump import to remote URLs. When using a public HTTP URL in /IndexImportMediawiki_p.html, the remote file now is directly streamed and processed, allowing import of several GB dumps even with a low memory remote peer, and without need to manually download the dump file first.
2017-04-14 14:32:44 +02:00
<a href="https://dumps.wikimedia.org/dewiki/latest/dewiki-latest-pages-articles.xml.bz2">
https://dumps.wikimedia.org/dewiki/latest/dewiki-latest-pages-articles.xml.bz2</a>.
Used a text input for wiki dump import file selection. Using an HTML "file" input was confusing (as reported by promocore on YaCy forum : http://forum.yacy-websuche.de/viewtopic.php?f=5&t=5965) , and it only worked with MS IE/Edge on a local YaCy peer : - for security reasons some current major browsers such as Firefox or Chrome do not allow to send full file path information when using a file form input - the local file system selection popup doesn't make sense when you want to import a dump on a remote YaCy server
2017-04-11 07:34:17 +02:00
</p>
<p>
Extended Mediawiki dump import to remote URLs. When using a public HTTP URL in /IndexImportMediawiki_p.html, the remote file now is directly streamed and processed, allowing import of several GB dumps even with a low memory remote peer, and without need to manually download the dump file first.
2017-04-14 14:32:44 +02:00
Dumps can be stored in the local file system or on a remote server in XML format and may be compressed in gz or bz2.
Used a text input for wiki dump import file selection. Using an HTML "file" input was confusing (as reported by promocore on YaCy forum : http://forum.yacy-websuche.de/viewtopic.php?f=5&t=5965) , and it only worked with MS IE/Edge on a local YaCy peer : - for security reasons some current major browsers such as Firefox or Chrome do not allow to send full file path information when using a file form input - the local file system selection popup doesn't make sense when you want to import a dump on a remote YaCy server
2017-04-11 07:34:17 +02:00
</p>
<div class="form-group">
<div class="col-sm-3 col-md-2 col-lg-2">
Extended Mediawiki dump import to remote URLs. When using a public HTTP URL in /IndexImportMediawiki_p.html, the remote file now is directly streamed and processed, allowing import of several GB dumps even with a low memory remote peer, and without need to manually download the dump file first.
2017-04-14 14:32:44 +02:00
<label for="file" class="control-label" >Dump file path or <abbr title="Uniform Resource Locator">URL</abbr></label>
Used a text input for wiki dump import file selection. Using an HTML "file" input was confusing (as reported by promocore on YaCy forum : http://forum.yacy-websuche.de/viewtopic.php?f=5&t=5965) , and it only worked with MS IE/Edge on a local YaCy peer : - for security reasons some current major browsers such as Firefox or Chrome do not allow to send full file path information when using a file form input - the local file system selection popup doesn't make sense when you want to import a dump on a remote YaCy server
2017-04-11 07:34:17 +02:00
</div>
<div class="col-sm-9 col-md-8 col-lg-8">
Extended Mediawiki dump import to remote URLs. When using a public HTTP URL in /IndexImportMediawiki_p.html, the remote file now is directly streamed and processed, allowing import of several GB dumps even with a low memory remote peer, and without need to manually download the dump file first.
2017-04-14 14:32:44 +02:00
<input id="file" class="form-control" name="file" type="text" title="Dump file path on this YaCy server file system, or any remote URL" required="required"/>
Used a text input for wiki dump import file selection. Using an HTML "file" input was confusing (as reported by promocore on YaCy forum : http://forum.yacy-websuche.de/viewtopic.php?f=5&t=5965) , and it only worked with MS IE/Edge on a local YaCy peer : - for security reasons some current major browsers such as Firefox or Chrome do not allow to send full file path information when using a file form input - the local file system selection popup doesn't make sense when you want to import a dump on a remote YaCy server
2017-04-11 07:34:17 +02:00
</div>
use input type=file for choosing IndexImportMediawiki_p dump file
2015-12-22 20:15:08 +01:00
</div>
Added MediaWiki dump import scheduling feature. Checking the last modified date by default to prevent unnecessary long running operations.
2017-05-03 18:53:01 +02:00
<div class="form-group">
<div class="col-sm-3">
<div class="checkbox">
<label>
<input name="iffresh" id="iffresh"
type="checkbox" checked="checked"
aria-describedby="iffreshInfo"/>
Import only when modified since last import
</label>
</div>
</div>
<div class="col-sm-5" id="iffreshInfo">
When checked, the dump file is imported only if its last modified date is unknown or is after the last import execution date on this same file
Fixed the previously added link to scheduled dump operations.
2017-05-04 08:45:30 +02:00
(see <a href="Table_API_p.html?filter=dump">recorded API calls</a> with the "dump" type).
Added MediaWiki dump import scheduling feature. Checking the last modified date by default to prevent unnecessary long running operations.
2017-05-03 18:53:01 +02:00
</div>
</div>
Used a text input for wiki dump import file selection. Using an HTML "file" input was confusing (as reported by promocore on YaCy forum : http://forum.yacy-websuche.de/viewtopic.php?f=5&t=5965) , and it only worked with MS IE/Edge on a local YaCy peer : - for security reasons some current major browsers such as Firefox or Chrome do not allow to send full file path information when using a file form input - the local file system selection popup doesn't make sense when you want to import a dump on a remote YaCy server
2017-04-11 07:34:17 +02:00
<input name="submit" class="btn btn-primary" type="submit" value="Import MediaWiki Dump" />
added a submenu to index administration to import a wikimedia dump (i.e. a dump from wikipedia) into the YaCy index: see http://localhost:8080/IndexImportWikimedia_p.html git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5930 6c8d7289-2bf4-0310-a012-ef5d649a1542
2009-05-08 09:54:10 +02:00
</fieldset>
</form>
<p>
When the import is started, the following happens:
enhanced the wikimedia dump import process enhanced the wiki parser and condenser speed git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5931 6c8d7289-2bf4-0310-a012-ef5d649a1542
2009-05-08 12:36:13 +02:00
</p><ul>
added a submenu to index administration to import a wikimedia dump (i.e. a dump from wikipedia) into the YaCy index: see http://localhost:8080/IndexImportWikimedia_p.html git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5930 6c8d7289-2bf4-0310-a012-ef5d649a1542
2009-05-08 09:54:10 +02:00
<li>The dump is extracted on the fly and wiki entries are translated into Dublin Core data format. The output looks like this:
<pre>
replaced utf-8 with UTF-8 git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7485 6c8d7289-2bf4-0310-a012-ef5d649a1542
2011-02-16 14:51:30 +01:00
&lt;?xml version="1.0" encoding="UTF-8"?&gt;
added a submenu to index administration to import a wikimedia dump (i.e. a dump from wikipedia) into the YaCy index: see http://localhost:8080/IndexImportWikimedia_p.html git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5930 6c8d7289-2bf4-0310-a012-ef5d649a1542
2009-05-08 09:54:10 +02:00
&lt;surrogates xmlns:dc="http://purl.org/dc/elements/1.1/"&gt;
&lt;record&gt;
&lt;dc:Title&gt;&lt;![CDATA[Alan Smithee]]&gt;&lt;/dc:Title&gt;
&lt;dc:Identifier&gt;http://de.wikipedia.org/wiki/Alan%20Smithee&lt;/dc:Identifier&gt;
&lt;dc:Description&gt;&lt;![CDATA[Der als Filmregisseur oft genannte Alan Smithee ist ein Anagramm]]&gt;&lt;/dc:Description&gt;
&lt;dc:Language&gt;de&lt;/dc:Language&gt;
&lt;dc:Date&gt;2009-05-07T06:03:48Z&lt;/dc:Date&gt;
&lt;/record&gt;
&lt;record&gt;
...
&lt;/record&gt;
&lt;/surrogates&gt;
</pre>
</li>
<li>Each 10000 wiki records are combined in one output file which is written to /DATA/SURROGATES/in into a temporary file.</li>
<li>When each of the generated output file is finished, it is renamed to a .xml file</li>
<li>Each time a xml surrogate file appears in /DATA/SURROGATES/in, the YaCy indexer fetches the file and indexes the record entries.</li>
<li>When a surrogate file is finished with indexing, it is moved to /DATA/SURROGATES/out</li>
<li>You can recycle processed surrogate files by moving them from /DATA/SURROGATES/out to /DATA/SURROGATES/in</li>
</ul>
de.lng: Added some more untranslated strings I found and uncommented old ones that were removed terminal_p.html: Put back the old ID which was really easy to find IndexCreate.js: Because XHTML 1.0 Strict does not allow name tags for some elements rewrote most element access functions to use getElementById Table_API_p.html and all other html pages: Some XHTMl 1.0 Strict fixes, changed checkAll javascript, marked the first row with checkboxes as unsortable where applicable Table_API_p.java and all other java pages: URLencoded lines with possible ampersands & -> &amp; for validation XHTML 1.0 Strict sourcecode --> All Index Create pages should validate now. Hope I did not break anything else (too much :-) git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7225 6c8d7289-2bf4-0310-a012-ef5d649a1542
2010-10-06 02:00:23 +02:00
<br />
added a submenu to index administration to import a wikimedia dump (i.e. a dump from wikipedia) into the YaCy index: see http://localhost:8080/IndexImportWikimedia_p.html git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5930 6c8d7289-2bf4-0310-a012-ef5d649a1542
2009-05-08 09:54:10 +02:00
::
Extended Mediawiki dump import to remote URLs. When using a public HTTP URL in /IndexImportMediawiki_p.html, the remote file now is directly streamed and processed, allowing import of several GB dumps even with a low memory remote peer, and without need to manually download the dump file first.
2017-04-14 14:32:44 +02:00
<p>#(status)#::<div class="alert alert-danger" role="alert">Error encountered : #[message]#</div>
#(/status)#</p>
enhanced the wikimedia dump import process enhanced the wiki parser and condenser speed git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5931 6c8d7289-2bf4-0310-a012-ef5d649a1542
2009-05-08 12:36:13 +02:00
<form><fieldset><legend>Import Process</legend>
added a submenu to index administration to import a wikimedia dump (i.e. a dump from wikipedia) into the YaCy index: see http://localhost:8080/IndexImportWikimedia_p.html git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5930 6c8d7289-2bf4-0310-a012-ef5d649a1542
2009-05-08 09:54:10 +02:00
<dl>
Improved MediaWiki dump import monitoring. When import thread is terminated : - now stop refreshing and stay on the monitoring page to give user a feedback after a long running import - added link to the next monitoring step : results from surrogates reader - added link to new import On the new import page, added a link on the eventual last import report.
2017-05-02 09:38:45 +02:00
<dt>Thread:</dt><dd>#(thread)#terminated. You can check the next step in the <a href="CrawlResults.html?process=7">Crawl Results page</a>
or start a <a href="IndexImportMediawiki_p.html">new import</a>.::started::running#(/thread)#</dd>
enhanced the wikimedia dump import process enhanced the wiki parser and condenser speed git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5931 6c8d7289-2bf4-0310-a012-ef5d649a1542
2009-05-08 12:36:13 +02:00
<dt>Dump:</dt><dd>#[dump]#</dd>
<dt>Processed:</dt><dd>#[count]# Wiki Entries</dd>
<dt>Speed:</dt><dd>#[speed]# articles per second</dd>
<dt>Running Time:</dt><dd>#[runningHours]# hours, #[runningMinutes]# minutes</dd>
<dt>Remaining Time:</dt><dd>#[remainingHours]# hours, #[remainingMinutes]# minutes</dd>
added a submenu to index administration to import a wikimedia dump (i.e. a dump from wikipedia) into the YaCy index: see http://localhost:8080/IndexImportWikimedia_p.html git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5930 6c8d7289-2bf4-0310-a012-ef5d649a1542
2009-05-08 09:54:10 +02:00
</dl>
enhanced the wikimedia dump import process enhanced the wiki parser and condenser speed git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5931 6c8d7289-2bf4-0310-a012-ef5d649a1542
2009-05-08 12:36:13 +02:00
</fieldset></form>
added a submenu to index administration to import a wikimedia dump (i.e. a dump from wikipedia) into the YaCy index: see http://localhost:8080/IndexImportWikimedia_p.html git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5930 6c8d7289-2bf4-0310-a012-ef5d649a1542
2009-05-08 09:54:10 +02:00
#(/import)#
#%env/templates/footer.template%#
</body>
</html>
Reference in New Issue Copy Permalink