jena-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
Subject svn commit: r931174 - in /websites/staging/jena/trunk/content: ./ documentation/hadoop/io.html
Date Mon, 01 Dec 2014 14:51:35 GMT
Author: buildbot
Date: Mon Dec  1 14:51:35 2014
New Revision: 931174

Staging update by buildbot for jena

    websites/staging/jena/trunk/content/   (props changed)

Propchange: websites/staging/jena/trunk/content/
--- cms:source-revision (original)
+++ cms:source-revision Mon Dec  1 14:51:35 2014
@@ -1 +1 @@

Modified: websites/staging/jena/trunk/content/documentation/hadoop/io.html
--- websites/staging/jena/trunk/content/documentation/hadoop/io.html (original)
+++ websites/staging/jena/trunk/content/documentation/hadoop/io.html Mon Dec  1 14:51:35 2014
@@ -157,7 +157,10 @@
 <li><a href="#output">Output</a><ul>
-<li><a href="#blank-nodes-in-output">Blank Nodes in Output</a></li>
+<li><a href="#blank-nodes-in-output">Blank Nodes in Output</a><ul>
+<li><a href="#blank-node-divergence-in-multi-stage-pipelines">Blank Node Divergence
in multi-stage pipelines</a></li>
 <li><a href="#node-output-format">Node Output Format</a></li>
@@ -166,16 +169,20 @@
 <li><a href="#output_1">Output</a></li>
-<li><a href="#configuration-options">Configuration Options</a><ul>
+<li><a href="#job-setup">Job Setup</a><ul>
+<li><a href="#job-configuration-options">Job Configuration Options</a><ul>
 <li><a href="#input-lines-per-batch">Input Lines per Batch</a></li>
 <li><a href="#max-line-length">Max Line Length</a></li>
 <li><a href="#ignoring-bad-tuples">Ignoring Bad Tuples</a></li>
+<li><a href="#global-blank-node-identity">Global Blank Node Identity</a></li>
 <li><a href="#output-batch-size">Output Batch Size</a></li>
 <h1 id="background-on-hadoop-io">Background on Hadoop IO</h1>
 <p>If you are already familiar with the Hadoop IO paradigm then please skip this section,
if not please read as otherwise some of the later information will not make much sense.</p>
@@ -184,8 +191,9 @@
 <p>In some cases there are file formats that may be processed in multiple ways i.e.
you can <em>split</em> them into pieces or you can process them as a whole.  Which
approach you wish to use will depend on whether you have a single file to process or many
files to process.  In the case of many files processing files as a whole may provide better
overall throughput than processing them as chunks.  However your mileage may vary especially
if your input data has many files of uneven size.</p>
 <h2 id="compressed-io">Compressed IO</h2>
 <p>Hadoop natively provides support for compressed input and output providing your
Hadoop cluster is appropriately configured.  The advantage of compressing the input/output
data is that it means there is less IO workload on the cluster however this comes with the
disadvantage that most compression formats block Hadoop's ability to <em>split</em>
up the input.</p>
+<p>Hadoop generally handles compression automatically and all our input and output
formats are capable of handling compressed input and output as necessary.</p>
 <h1 id="rdf-io-in-hadoop">RDF IO in Hadoop</h1>
-<p>There are a wide range of RDF serialisations supported by ARQ, please see the <a
href="../io/">RDF IO</a> for an overview of the formats that Jena supports.</p>
+<p>There are a wide range of RDF serialisations supported by ARQ, please see the <a
href="../io/">RDF IO</a> for an overview of the formats that Jena supports.  In this
section we go into a lot more depth of how exactly we support RDF IO in Hadoop.</p>
 <h2 id="input">Input</h2>
 <p>One of the difficulties posed when wrapping these for Hadoop IO is that the formats
have very different properties in terms of our ability to <em>split</em> them
into distinct chunks for Hadoop to process.  So we categorise the possible ways to process
RDF inputs as follows:</p>
@@ -213,6 +221,10 @@
 <p>As with input blank nodes provide a complicating factor in producing RDF output.
 For whole file output formats this is not an issue but it does need to be considered for
line and batch based formats.</p>
 <p>However what we have found in practise is that the Jena writers will predictably
map internal identifiers to the blank node identifiers in the output serialisations.  What
this means is that even when processing output in batches we've found that using the line/batch
based formats correctly preserve blank node identity.</p>
 <p>If you are concerned about potential data corruption as a result of this then you
should make sure to always choose a whole file output format but be aware that this can exhaust
memory if your output is large.</p>
+<h4 id="blank-node-divergence-in-multi-stage-pipelines">Blank Node Divergence in multi-stage
+<p>The other thing to consider with regards to blank nodes in output is that Hadoop
will by default create multiple output files (one for each reducer) so even if consistent
and valid blank nodes are output they may be spread over multiple files.</p>
+<p>In multi-stage pipelines you will need to manually concatenate these files back
together (assuming they are in a format that allows this e.g. NTriples) as otherwise when
you pass them as input to the next job the blank node identifiers will diverge from each other.
 <a href="">JENA-820</a> discusses
this problem and introduces a special configuration setting that can be used to resolve this.
 Note that even with this setting enabled some formats are not capable of respecting it, see
the later section on <a href="#job-configuration-options">Job Configuration Options</a>
for more details.</p>
+<p>An alternative workaround is to always use RDF Thrift as the intermediate output
format since it preserves blank node identifiers precisely as they are seen.  This also has
the advantage that RDF Thrift is extremely fast to read and write which can speed up multi-stage
pipelines considerably.</p>
 <h3 id="node-output-format">Node Output Format</h3>
 <p>We also include a special <code>NTriplesNodeOutputFormat</code> which
is capable of outputting pairs composed of a <code>NodeWritable</code> key and
any value type.  Think of this as being similar to the standard Hadoop <code>TextOutputFormat</code>
except it understands how to format nodes as valid NTriples serialisation.  This format is
useful when performing simple statistical analysis such as node usage counts or other calculations
over nodes.</p>
 <p>In the case where the value of the key value pair is also a RDF primitive proper
NTriples formatting is also applied to each of the nodes in the value</p>
@@ -275,9 +287,28 @@
   <tr><td>RDF Thrift</td><td>Yes</td><td>No</td><td>No</td></tr>
-<h2 id="configuration-options">Configuration Options</h2>
+<h2 id="job-setup">Job Setup</h2>
+<p>To use RDF as an input and/or output format you will need to configure your Job
appropriately, this requires setting the input/output format and setting the data paths:</p>
+<div class="codehilite"><pre><span class="c1">// Create a job using default
+<span class="n">Job</span> <span class="n">job</span> <span class="o">=</span>
<span class="n">Job</span><span class="p">.</span><span class="n">createInstance</span><span
class="p">(</span><span class="k">new</span> <span class="n">Configuration</span><span
class="p">(</span><span class="n">true</span><span class="p">));</span>
+<span class="c1">// Use Turtle as the input format</span>
+<span class="n">job</span><span class="p">.</span><span class="n">setInputFormatClass</span><span
class="p">(</span><span class="n">TurtleInputFormat</span><span class="p">.</span><span
class="k">class</span><span class="p">);</span>
+<span class="n">FileInputFormat</span><span class="p">.</span><span
class="n">setInputPath</span><span class="p">(</span><span class="n">job</span><span
class="p">,</span> <span class="s">&quot;/users/example/input&quot;</span><span
+<span class="c1">// Use NTriples as the output format</span>
+<span class="n">job</span><span class="p">.</span><span class="n">setOutputFormatClass</span><span
class="p">(</span><span class="n">NTriplesOutputFormat</span><span
class="p">.</span><span class="k">class</span><span class="p">);</span>
+<span class="n">FileOutputFormat</span><span class="p">.</span><span
class="n">setOutputPath</span><span class="p">(</span><span class="n">job</span><span
class="p">,</span> <span class="s">&quot;/users/example/output&quot;</span><span
+<span class="c1">// Other job configuration...</span>
+<p>This example takes in input in Turtle format from the directory <code>/users/example/input</code>
and outputs the end results in NTriples in the directory <code>/users/example/output</code>.</p>
+<p>Take a look at the <a href="../javadoc/hadoop/io/">Javadocs</a> to find
the actual available input and output format implementations.</p>
+<h3 id="job-configuration-options">Job Configuration Options</h3>
 <p>There are a several useful configuration options that can be used to tweak the behaviour
of the RDF IO functionality if desired.</p>
-<h3 id="input-lines-per-batch">Input Lines per Batch</h3>
+<h4 id="input-lines-per-batch">Input Lines per Batch</h4>
 <p>Since our line based input formats use the standard Hadoop <code>NLineInputFormat</code>
to decide how to split up inputs we support the standard <code>mapreduce.input.lineinputformat.linespermap</code>
configuration setting for changing the number of lines processed per map.</p>
 <p>You can set this directly in your configuration:</p>
 <div class="codehilite"><pre><span class="n">job</span><span class="p">.</span><span
class="n">getConfiguration</span><span class="p">().</span><span class="n">setInt</span><span
class="p">(</span><span class="n">NLineInputFormat</span><span class="p">.</span><span
class="n">LINES_PER_MAP</span><span class="p">,</span> 100<span class="p">);</span>
@@ -289,19 +320,28 @@
-<h3 id="max-line-length">Max Line Length</h3>
+<h4 id="max-line-length">Max Line Length</h4>
 <p>When using line based inputs it may be desirable to ignore lines that exceed a certain
length (for example if you are not interested in really long literals).  Again we use the
standard Hadoop configuration setting <code>mapreduce.input.linerecordreader.line.maxlength</code>
to control this behaviour:</p>
 <div class="codehilite"><pre><span class="n">job</span><span class="p">.</span><span
class="n">getConfiguration</span><span class="p">().</span><span class="n">setInt</span><span
class="p">(</span><span class="n">HadoopIOConstants</span><span class="p">.</span><span
class="n">MAX_LINE_LENGTH</span><span class="p">,</span> 8192<span
-<h3 id="ignoring-bad-tuples">Ignoring Bad Tuples</h3>
+<h4 id="ignoring-bad-tuples">Ignoring Bad Tuples</h4>
 <p>In many cases you may have data that you know contains invalid tuples, in such cases
it can be useful to just ignore the bad tuples and continue.  By default we enable this behaviour
and will skip over bad tuples though they will be logged as an error.  If you want you can
disable this behaviour by setting the <code></code>
configuration setting:</p>
 <div class="codehilite"><pre><span class="n">job</span><span class="p">.</span><span
class="n">getConfiguration</span><span class="p">().</span><span class="n">setBoolean</span><span
class="p">(</span><span class="n">RdfIOConstants</span><span class="p">.</span><span
class="n">INPUT_IGNORE_BAD_TUPLES</span><span class="p">,</span> <span
class="n">false</span><span class="p">);</span>
-<h3 id="output-batch-size">Output Batch Size</h3>
+<h4 id="global-blank-node-identity">Global Blank Node Identity</h4>
+<p>The default behaviour of these libraries is to allocate file scoped blank node identifiers
in such a way that the same syntactic identifier read from the same file (even if by different
nodes/processes) is allocated the same blank node ID.  However the same syntactic identifier
in different files should result in different blank nodes.  However as discussed earlier in
the case of multi-stage jobs the intermediate outputs may be split over several files which
can cause the blank node identifiers to diverge from each other when they are read back in.</p>
+<p>For multi-stage jobs this is often (but not always) incorrect and undesirable behaviour
in which case you can set the <code></code>
property to true:</p>
+<div class="codehilite"><pre><span class="n">job</span><span class="p">.</span><span
class="n">getConfiguration</span><span class="p">.</span><span class="n">setBoolean</span><span
class="p">(</span><span class="n">RdfIOConstants</span><span class="p">.</span><span
class="n">GLOBAL_BNODE_IDENTITY</span><span class="p">,</span> <span
class="n">true</span><span class="p">);</span>
+<p>Note however that not all formats are capable of honouring this option, notably
RDF/XML and JSON-LD.</p>
+<p>As noted earlier an alternative workaround is to use RDF Thrift as the intermediate
format since it guarantees to preserve blank node identifiers precisely.</p>
+<h4 id="output-batch-size">Output Batch Size</h4>
 <p>The batch size for batched output formats can be controlled by setting the <code></code>
property as desired.  The default value for this if not explicitly configured is 10,000:</p>
 <div class="codehilite"><pre><span class="n">job</span><span class="p">.</span><span
class="n">getConfiguration</span><span class="p">.</span><span class="n">setInt</span><span
class="p">(</span><span class="n">RdfIOConstants</span><span class="p">.</span><span
class="n">OUTPUT_BATCH_SIZE</span><span class="p">,</span> 25000<span

View raw message