Wikimedia Dump Import

#(status)#No import thread is running, you can start a new thread here::Bad input data: #[message]# #(/status)#

When the import is started, the following happens:

• The dump is extracted on the fly and wiki entries are translated into Dublin Core data format. The output looks like this:

      <?xml version="1.0" encoding="utf-8"?>
  <surrogates xmlns:dc="http://purl.org/dc/elements/1.1/">
    <record>
      <dc:Title><![CDATA[Alan Smithee]]></dc:Title>
      <dc:Identifier>http://de.wikipedia.org/wiki/Alan%20Smithee</dc:Identifier>
      <dc:Description><![CDATA[Der als Filmregisseur oft genannte Alan Smithee ist ein Anagramm]]></dc:Description>
      <dc:Language>de</dc:Language>
      <dc:Date>2009-05-07T06:03:48Z</dc:Date>
    </record>
    <record>
      ...
    </record> 
  </surrogates>
      

• Each 10000 wiki records are combined in one output file which is written to /DATA/SURROGATES/in into a temporary file.
• When each of the generated output file is finished, it is renamed to a .xml file
• Each time a xml surrogate file appears in /DATA/SURROGATES/in, the YaCy indexer fetches the file and indexes the record entries.
• When a surrogate file is finished with indexing, it is moved to /DATA/SURROGATES/out
• You can recycle processed surrogate files by moving them from /DATA/SURROGATES/out to /DATA/SURROGATES/in