beam-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From al...@apache.org
Subject [2/3] beam-site git commit: Regenerate website
Date Wed, 03 May 2017 00:59:46 GMT
Regenerate website


Project: http://git-wip-us.apache.org/repos/asf/beam-site/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam-site/commit/5a7282fe
Tree: http://git-wip-us.apache.org/repos/asf/beam-site/tree/5a7282fe
Diff: http://git-wip-us.apache.org/repos/asf/beam-site/diff/5a7282fe

Branch: refs/heads/asf-site
Commit: 5a7282fe42e9a396673aacfac8f466284dffa1ea
Parents: 6289505
Author: Ahmet Altay <altay@altay-macbookpro2.roam.corp.google.com>
Authored: Tue May 2 17:59:11 2017 -0700
Committer: Ahmet Altay <altay@altay-macbookpro2.roam.corp.google.com>
Committed: Tue May 2 17:59:11 2017 -0700

----------------------------------------------------------------------
 content/documentation/io/built-in/index.html    |  2 +-
 .../documentation/programming-guide/index.html  | 46 +++++++++++++++++---
 .../sdks/python-type-safety/index.html          | 10 ++---
 3 files changed, 45 insertions(+), 13 deletions(-)
----------------------------------------------------------------------


http://git-wip-us.apache.org/repos/asf/beam-site/blob/5a7282fe/content/documentation/io/built-in/index.html
----------------------------------------------------------------------
diff --git a/content/documentation/io/built-in/index.html b/content/documentation/io/built-in/index.html
index 639a877..cf262d3 100644
--- a/content/documentation/io/built-in/index.html
+++ b/content/documentation/io/built-in/index.html
@@ -170,7 +170,7 @@
   <td>Java</td>
   <td>
     <p><a href="https://github.com/apache/beam/blob/master/sdks/java/core/src/main/java/org/apache/beam/sdk/io/AvroIO.java">AvroIO</a></p>
-    <p><a href="https://github.com/apache/beam/tree/master/sdks/java/io/hdfs">Apache
Hadoop HDFS</a></p>
+    <p><a href="https://github.com/apache/beam/tree/master/sdks/java/io/hadoop-file-system">Apache
Hadoop File System</a></p>
     <p><a href="https://github.com/apache/beam/blob/master/sdks/java/core/src/main/java/org/apache/beam/sdk/io/TextIO.java">TextIO</a></p>
     <p><a href="https://github.com/apache/beam/blob/master/sdks/java/core/src/main/java/org/apache/beam/sdk/io/">XML</a></p>
   </td>

http://git-wip-us.apache.org/repos/asf/beam-site/blob/5a7282fe/content/documentation/programming-guide/index.html
----------------------------------------------------------------------
diff --git a/content/documentation/programming-guide/index.html b/content/documentation/programming-guide/index.html
index b57f89c..5ad4bea 100644
--- a/content/documentation/programming-guide/index.html
+++ b/content/documentation/programming-guide/index.html
@@ -1200,18 +1200,18 @@ guest, [[], [order4]]
       <span class="k">yield</span> <span class="n">element</span>
     <span class="k">else</span><span class="p">:</span>
       <span class="c"># Emit this word's long length to the 'above_cutoff_lengths'
output.</span>
-      <span class="k">yield</span> <span class="n">pvalue</span><span
class="o">.</span><span class="n">OutputValue</span><span class="p">(</span>
+      <span class="k">yield</span> <span class="n">pvalue</span><span
class="o">.</span><span class="n">TaggedOutput</span><span class="p">(</span>
           <span class="s">'above_cutoff_lengths'</span><span class="p">,</span>
<span class="nb">len</span><span class="p">(</span><span class="n">element</span><span
class="p">))</span>
     <span class="k">if</span> <span class="n">element</span><span
class="o">.</span><span class="n">startswith</span><span class="p">(</span><span
class="n">marker</span><span class="p">):</span>
       <span class="c"># Emit this word to a different output with the 'marked strings'
tag.</span>
-      <span class="k">yield</span> <span class="n">pvalue</span><span
class="o">.</span><span class="n">OutputValue</span><span class="p">(</span><span
class="s">'marked strings'</span><span class="p">,</span> <span class="n">element</span><span
class="p">)</span>
+      <span class="k">yield</span> <span class="n">pvalue</span><span
class="o">.</span><span class="n">TaggedOutput</span><span class="p">(</span><span
class="s">'marked strings'</span><span class="p">,</span> <span class="n">element</span><span
class="p">)</span>
 
 
 <span class="c"># Producing multiple outputs is also available in Map and FlatMap.</span>
 <span class="c"># Here is an example that uses FlatMap and shows that the tags do not
need to be specified ahead of time.</span>
 
 <span class="k">def</span> <span class="nf">even_odd</span><span
class="p">(</span><span class="n">x</span><span class="p">):</span>
-  <span class="k">yield</span> <span class="n">pvalue</span><span
class="o">.</span><span class="n">OutputValue</span><span class="p">(</span><span
class="s">'odd'</span> <span class="k">if</span> <span class="n">x</span>
<span class="o">%</span> <span class="mi">2</span> <span class="k">else</span>
<span class="s">'even'</span><span class="p">,</span> <span class="n">x</span><span
class="p">)</span>
+  <span class="k">yield</span> <span class="n">pvalue</span><span
class="o">.</span><span class="n">TaggedOutput</span><span class="p">(</span><span
class="s">'odd'</span> <span class="k">if</span> <span class="n">x</span>
<span class="o">%</span> <span class="mi">2</span> <span class="k">else</span>
<span class="s">'even'</span><span class="p">,</span> <span class="n">x</span><span
class="p">)</span>
   <span class="k">if</span> <span class="n">x</span> <span class="o">%</span>
<span class="mi">10</span> <span class="o">==</span> <span class="mi">0</span><span
class="p">:</span>
     <span class="k">yield</span> <span class="n">x</span>
 
@@ -1267,7 +1267,17 @@ guest, [[], [order4]]
 </code></pre>
 </div>
 
-<div class="language-py highlighter-rouge"><pre class="highlight"><code>
 <span class="n">Python</span> <span class="n">code</span> <span
class="n">snippet</span> <span class="n">coming</span> <span class="n">soon</span>
<span class="p">(</span><span class="n">BEAM</span><span class="o">-</span><span
class="mi">1926</span><span class="p">)</span>
+<div class="language-py highlighter-rouge"><pre class="highlight"><code><span
class="c"># The CountWords Composite Transform inside the WordCount pipeline.</span>
+<span class="k">class</span> <span class="nc">CountWords</span><span
class="p">(</span><span class="n">beam</span><span class="o">.</span><span
class="n">PTransform</span><span class="p">):</span>
+
+  <span class="k">def</span> <span class="nf">expand</span><span
class="p">(</span><span class="bp">self</span><span class="p">,</span>
<span class="n">pcoll</span><span class="p">):</span>
+    <span class="k">return</span> <span class="p">(</span><span
class="n">pcoll</span>
+            <span class="c"># Convert lines of text into individual words.</span>
+            <span class="o">|</span> <span class="s">'ExtractWords'</span>
<span class="o">&gt;&gt;</span> <span class="n">beam</span><span
class="o">.</span><span class="n">ParDo</span><span class="p">(</span><span
class="n">ExtractWordsFn</span><span class="p">())</span>
+            <span class="c"># Count the number of times each word occurs.</span>
+            <span class="o">|</span> <span class="n">beam</span><span
class="o">.</span><span class="n">combiners</span><span class="o">.</span><span
class="n">Count</span><span class="o">.</span><span class="n">PerElement</span><span
class="p">()</span>
+            <span class="c"># Format each word and count into a printable string.</span>
+            <span class="o">|</span> <span class="s">'FormatCounts'</span>
<span class="o">&gt;&gt;</span> <span class="n">beam</span><span
class="o">.</span><span class="n">ParDo</span><span class="p">(</span><span
class="n">FormatCountsFn</span><span class="p">()))</span>
 </code></pre>
 </div>
 
@@ -1286,7 +1296,10 @@ guest, [[], [order4]]
 </code></pre>
 </div>
 
-<div class="language-py highlighter-rouge"><pre class="highlight"><code>
 <span class="n">Python</span> <span class="n">code</span> <span
class="n">snippet</span> <span class="n">coming</span> <span class="n">soon</span>
<span class="p">(</span><span class="n">BEAM</span><span class="o">-</span><span
class="mi">1926</span><span class="p">)</span>
+<div class="language-py highlighter-rouge"><pre class="highlight"><code><span
class="k">class</span> <span class="nc">ComputeWordLengths</span><span
class="p">(</span><span class="n">beam</span><span class="o">.</span><span
class="n">PTransform</span><span class="p">):</span>
+  <span class="k">def</span> <span class="nf">expand</span><span
class="p">(</span><span class="bp">self</span><span class="p">,</span>
<span class="n">pcoll</span><span class="p">):</span>
+    <span class="c"># transform logic goes here</span>
+    <span class="k">return</span> <span class="n">pcoll</span> <span
class="o">|</span> <span class="n">beam</span><span class="o">.</span><span
class="n">Map</span><span class="p">(</span><span class="k">lambda</span>
<span class="n">x</span><span class="p">:</span> <span class="nb">len</span><span
class="p">(</span><span class="n">x</span><span class="p">))</span>
 </code></pre>
 </div>
 
@@ -1307,7 +1320,10 @@ guest, [[], [order4]]
 </code></pre>
 </div>
 
-<div class="language-py highlighter-rouge"><pre class="highlight"><code>
 <span class="n">Python</span> <span class="n">code</span> <span
class="n">snippet</span> <span class="n">coming</span> <span class="n">soon</span>
<span class="p">(</span><span class="n">BEAM</span><span class="o">-</span><span
class="mi">1926</span><span class="p">)</span>
+<div class="language-py highlighter-rouge"><pre class="highlight"><code><span
class="k">class</span> <span class="nc">ComputeWordLengths</span><span
class="p">(</span><span class="n">beam</span><span class="o">.</span><span
class="n">PTransform</span><span class="p">):</span>
+  <span class="k">def</span> <span class="nf">expand</span><span
class="p">(</span><span class="bp">self</span><span class="p">,</span>
<span class="n">pcoll</span><span class="p">):</span>
+    <span class="c"># transform logic goes here</span>
+    <span class="k">return</span> <span class="n">pcoll</span> <span
class="o">|</span> <span class="n">beam</span><span class="o">.</span><span
class="n">Map</span><span class="p">(</span><span class="k">lambda</span>
<span class="n">x</span><span class="p">:</span> <span class="nb">len</span><span
class="p">(</span><span class="n">x</span><span class="p">))</span>
 </code></pre>
 </div>
 
@@ -1920,7 +1936,7 @@ Subsequent transforms, however, are applied to the result of the <code
class="hi
 </code></pre>
 </div>
 
-<div class="language-py highlighter-rouge"><pre class="highlight"><code>
 <span class="n">The</span> <span class="n">Beam</span> <span class="n">SDK</span>
<span class="k">for</span> <span class="n">Python</span> <span
class="n">does</span> <span class="ow">not</span> <span class="n">support</span>
<span class="n">triggers</span><span class="o">.</span>
+<div class="language-py highlighter-rouge"><pre class="highlight"><code>
 <span class="c"># The Beam SDK for Python does not support triggers.</span>
 </code></pre>
 </div>
 
@@ -1956,6 +1972,10 @@ Subsequent transforms, however, are applied to the result of the <code
class="hi
 </code></pre>
 </div>
 
+<div class="language-py highlighter-rouge"><pre class="highlight"><code>
 <span class="c"># The Beam SDK for Python does not support triggers.</span>
+</code></pre>
+</div>
+
 <p>This code sample sets a time-based trigger for a <code class="highlighter-rouge">PCollection</code>,
which emits results one minute after the first element in that window has been processed.
The last line in the code sample, <code class="highlighter-rouge">.discardingFiredPanes()</code>,
is the window’s <strong>accumulation mode</strong>.</p>
 
 <h4 id="window-accumulation-modes">Window Accumulation Modes</h4>
@@ -2009,6 +2029,10 @@ Subsequent transforms, however, are applied to the result of the <code
class="hi
 </code></pre>
 </div>
 
+<div class="language-py highlighter-rouge"><pre class="highlight"><code>
 <span class="c"># The Beam SDK for Python does not support triggers.</span>
+</code></pre>
+</div>
+
 <p>This allowed lateness propagates to all <code class="highlighter-rouge">PCollection</code>s
derived as a result of applying transforms to the original <code class="highlighter-rouge">PCollection</code>.
If you want to change the allowed lateness later in your pipeline, you can apply <code
class="highlighter-rouge">Window.withAllowedLateness()</code> again, explicitly.</p>
 
 <h3 id="a-namecomposite-triggersacomposite-triggers"><a name="composite-triggers"></a>Composite
Triggers</h3>
@@ -2055,6 +2079,10 @@ Subsequent transforms, however, are applied to the result of the <code
class="hi
 </code></pre>
 </div>
 
+<div class="language-py highlighter-rouge"><pre class="highlight"><code>
 <span class="c"># The Beam SDK for Python does not support triggers.</span>
+</code></pre>
+</div>
+
 <h4 id="other-composite-triggers">Other Composite Triggers</h4>
 
 <p>You can also build other sorts of composite triggers. The following example code
shows a simple composite trigger that fires whenever the pane has at least 100 elements, or
after a minute.</p>
@@ -2065,6 +2093,10 @@ Subsequent transforms, however, are applied to the result of the <code
class="hi
 </code></pre>
 </div>
 
+<div class="language-py highlighter-rouge"><pre class="highlight"><code>
 <span class="c"># The Beam SDK for Python does not support triggers.</span>
+</code></pre>
+</div>
+
 
       </div>
 

http://git-wip-us.apache.org/repos/asf/beam-site/blob/5a7282fe/content/documentation/sdks/python-type-safety/index.html
----------------------------------------------------------------------
diff --git a/content/documentation/sdks/python-type-safety/index.html b/content/documentation/sdks/python-type-safety/index.html
index 24900b8..cd364cf 100644
--- a/content/documentation/sdks/python-type-safety/index.html
+++ b/content/documentation/sdks/python-type-safety/index.html
@@ -167,7 +167,9 @@
 
 <p>Consider the following example, in which <code class="highlighter-rouge">numbers</code>
is a <code class="highlighter-rouge">PCollection</code> of <code class="highlighter-rouge">str</code>
values:</p>
 
-<div class="highlighter-rouge"><pre class="highlight"><code>numbers = p
| beam.Create(['1', '2', '3'])
+<div class="highlighter-rouge"><pre class="highlight"><code>p = TestPipeline(options=PipelineOptions(pipeline_type_check=True))
+
+numbers = p | beam.Create(['1', '2', '3'])
 </code></pre>
 </div>
 
@@ -189,8 +191,7 @@
 
 <p>To specify type hints inline, use the methods <code class="highlighter-rouge">with_input_types</code>
and <code class="highlighter-rouge">with_output_types</code>. The following example
code declares an input type hint inline:</p>
 
-<div class="highlighter-rouge"><pre class="highlight"><code>p.options.view_as(TypeOptions).pipeline_type_check
= True
-evens = numbers | beam.Filter(lambda x: x % 2 == 0).with_input_types(int)
+<div class="highlighter-rouge"><pre class="highlight"><code>evens = numbers
| beam.Filter(lambda x: x % 2 == 0).with_input_types(int)
 </code></pre>
 </div>
 
@@ -284,8 +285,7 @@ p.run()
 
 <p>However, if you enable runtime type checking, the code passes at pipeline construction
and fails at runtime. To enable runtime type checking, set the pipeline option <code class="highlighter-rouge">runtime_type_check</code>
to <code class="highlighter-rouge">True</code>.</p>
 
-<div class="highlighter-rouge"><pre class="highlight"><code>p.options.view_as(TypeOptions).runtime_type_check
= True
-p | beam.Create(['a']) | beam.Map(lambda x: 3).with_output_types(str)
+<div class="highlighter-rouge"><pre class="highlight"><code>p | beam.Create(['a'])
| beam.Map(lambda x: 3).with_output_types(str)
 p.run()
 </code></pre>
 </div>


Mime
View raw message