beam-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From git-site-r...@apache.org
Subject [beam] branch asf-site updated: Publishing website 2021/07/27 18:01:39 at commit 69822e4
Date Tue, 27 Jul 2021 18:02:09 GMT
This is an automated email from the ASF dual-hosted git repository.

git-site-role pushed a commit to branch asf-site
in repository https://gitbox.apache.org/repos/asf/beam.git


The following commit(s) were added to refs/heads/asf-site by this push:
     new 7fcfe6a  Publishing website 2021/07/27 18:01:39 at commit 69822e4
7fcfe6a is described below

commit 7fcfe6a62e375a661d76c0b1fe0159f0c1a15471
Author: jenkins <builds@apache.org>
AuthorDate: Tue Jul 27 18:01:40 2021 +0000

    Publishing website 2021/07/27 18:01:39 at commit 69822e4
---
 website/generated-content/documentation/index.xml  | 34 ++++++++++++++++++++++
 .../documentation/programming-guide/index.html     | 12 ++++----
 website/generated-content/sitemap.xml              |  2 +-
 3 files changed, 42 insertions(+), 6 deletions(-)

diff --git a/website/generated-content/documentation/index.xml b/website/generated-content/documentation/index.xml
index 28f2ae0..8690ebb 100644
--- a/website/generated-content/documentation/index.xml
+++ b/website/generated-content/documentation/index.xml
@@ -6624,6 +6624,14 @@ transform at any point while constructing your pipeline to create a
new
 &lt;div class="highlight">&lt;pre class="chroma">&lt;code class="language-py"
data-lang="py">&lt;span class="n">lines&lt;/span> &lt;span class="o">=&lt;/span>
&lt;span class="n">pipeline&lt;/span> &lt;span class="o">|&lt;/span>
&lt;span class="n">beam&lt;/span>&lt;span class="o">.&lt;/span>&lt;span
class="n">io&lt;/span>&lt;span class="o">.&lt;/span>&lt;span class="n">ReadFromText&lt;/span>&lt;span
class="p">(&lt;/span>&lt;span class="s1">&amp;#39;gs://some/inputData.txt&amp;#39;&lt;/spa
[...]
 &lt;/div>
 &lt;/div>
+&lt;div class='language-go snippet'>
+&lt;div class="notebook-skip code-snippet">
+&lt;a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom"
title="Copy to clipboard">
+&lt;img src="/images/copy-icon.svg"/>
+&lt;/a>
+&lt;div class="highlight">&lt;pre class="chroma">&lt;code class="language-go"
data-lang="go">&lt;span class="nx">lines&lt;/span> &lt;span class="o">:=&lt;/span>
&lt;span class="nx">textio&lt;/span>&lt;span class="p">.&lt;/span>&lt;span
class="nf">Read&lt;/span>&lt;span class="p">(&lt;/span>&lt;span
class="nx">scope&lt;/span>&lt;span class="p">,&lt;/span> &lt;span
class="err">&amp;#39;&lt;/span>&lt;span class="nx">gs&lt;/span>&lt;span
class="p">:&lt;/span>&lt;span class="o">//&lt;/span>&l [...]
+&lt;/div>
+&lt;/div>
 &lt;h3 id="pipeline-io-writing-data">5.2. Writing output data&lt;/h3>
 &lt;p>Write transforms write the data in a &lt;code>PCollection&lt;/code>
to an external data source.
 You will most often use write transforms at the end of your pipeline to output
@@ -6645,6 +6653,14 @@ a &lt;code>PCollection&lt;/code>'s data at any point
in your pipeline.&lt;/p>
 &lt;div class="highlight">&lt;pre class="chroma">&lt;code class="language-py"
data-lang="py">&lt;span class="n">output&lt;/span> &lt;span class="o">|&lt;/span>
&lt;span class="n">beam&lt;/span>&lt;span class="o">.&lt;/span>&lt;span
class="n">io&lt;/span>&lt;span class="o">.&lt;/span>&lt;span class="n">WriteToText&lt;/span>&lt;span
class="p">(&lt;/span>&lt;span class="s1">&amp;#39;gs://some/outputData&amp;#39;&lt;/span>&lt;span
class="p">)&lt;/span>&lt;/code>&lt;/pre>&lt;/div>
 &lt;/div>
 &lt;/div>
+&lt;div class='language-go snippet'>
+&lt;div class="notebook-skip code-snippet">
+&lt;a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom"
title="Copy to clipboard">
+&lt;img src="/images/copy-icon.svg"/>
+&lt;/a>
+&lt;div class="highlight">&lt;pre class="chroma">&lt;code class="language-go"
data-lang="go">&lt;span class="nx">textio&lt;/span>&lt;span class="p">.&lt;/span>&lt;span
class="nf">Write&lt;/span>&lt;span class="p">(&lt;/span>&lt;span
class="nx">scope&lt;/span>&lt;span class="p">,&lt;/span> &lt;span
class="err">&amp;#39;&lt;/span>&lt;span class="nx">gs&lt;/span>&lt;span
class="p">:&lt;/span>&lt;span class="o">//&lt;/span>&lt;span class="nx">some&lt;/span>&lt;span
class="o">/&lt;/span>&lt;s [...]
+&lt;/div>
+&lt;/div>
 &lt;h3 id="file-based-data">5.3. File-based input and output data&lt;/h3>
 &lt;h4 id="file-based-reading-multiple-locations">5.3.1. Reading from multiple locations&lt;/h4>
 &lt;p>Many read transforms support reading from multiple input files matching a glob
@@ -6670,6 +6686,14 @@ suffix &amp;ldquo;.csv&amp;rdquo; in the given location:&lt;/p>
 &lt;span class="s1">&amp;#39;path/to/input-*.csv&amp;#39;&lt;/span>&lt;span
class="p">)&lt;/span>&lt;/code>&lt;/pre>&lt;/div>
 &lt;/div>
 &lt;/div>
+&lt;div class='language-go snippet'>
+&lt;div class="notebook-skip code-snippet">
+&lt;a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom"
title="Copy to clipboard">
+&lt;img src="/images/copy-icon.svg"/>
+&lt;/a>
+&lt;div class="highlight">&lt;pre class="chroma">&lt;code class="language-go"
data-lang="go">&lt;span class="nx">lines&lt;/span> &lt;span class="o">:=&lt;/span>
&lt;span class="nx">textio&lt;/span>&lt;span class="p">.&lt;/span>&lt;span
class="nf">Read&lt;/span>&lt;span class="p">(&lt;/span>&lt;span
class="nx">scope&lt;/span>&lt;span class="p">,&lt;/span> &lt;span
class="s">&amp;#34;path/to/input-*.csv&amp;#34;&lt;/span>&lt;span class="p">)&lt;/span>&lt;/code>&lt;/pre>&lt;/div>
+&lt;/div>
+&lt;/div>
 &lt;p>To read data from disparate sources into a single &lt;code>PCollection&lt;/code>,
read each one
 independently and then use the &lt;a href="#flatten">Flatten&lt;/a> transform
to create a single
 &lt;code>PCollection&lt;/code>.&lt;/p>
@@ -6700,6 +6724,16 @@ location. Each file has the prefix &amp;ldquo;numbers&amp;rdquo;,
a numeric tag,
 &lt;span class="s1">&amp;#39;/path/to/numbers&amp;#39;&lt;/span>&lt;span
class="p">,&lt;/span> &lt;span class="n">file_name_suffix&lt;/span>&lt;span
class="o">=&lt;/span>&lt;span class="s1">&amp;#39;.csv&amp;#39;&lt;/span>&lt;span
class="p">)&lt;/span>&lt;/code>&lt;/pre>&lt;/div>
 &lt;/div>
 &lt;/div>
+&lt;div class='language-go snippet'>
+&lt;div class="notebook-skip code-snippet">
+&lt;a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom"
title="Copy to clipboard">
+&lt;img src="/images/copy-icon.svg"/>
+&lt;/a>
+&lt;div class="highlight">&lt;pre class="chroma">&lt;code class="language-go"
data-lang="go">&lt;span class="c1">// The Go SDK textio doesn&amp;#39;t support
sharding on writes yet.
+&lt;/span>&lt;span class="c1">// See https://issues.apache.org/jira/browse/BEAM-12664
for ways
+&lt;/span>&lt;span class="c1">&lt;/span>&lt;span class="o">//&lt;/span>
&lt;span class="nx">to&lt;/span> &lt;span class="nx">contribute&lt;/span>
&lt;span class="nx">a&lt;/span> &lt;span class="nx">solution&lt;/span>&lt;span
class="p">.&lt;/span>&lt;/code>&lt;/pre>&lt;/div>
+&lt;/div>
+&lt;/div>
 &lt;h3 id="provided-io-transforms">5.4. Beam-provided I/O transforms&lt;/h3>
 &lt;p>See the &lt;a href="/documentation/io/built-in/">Beam-provided I/O Transforms&lt;/a>
 page for a list of the currently available I/O transforms.&lt;/p>
diff --git a/website/generated-content/documentation/programming-guide/index.html b/website/generated-content/documentation/programming-guide/index.html
index 589910c..1e327ad 100644
--- a/website/generated-content/documentation/programming-guide/index.html
+++ b/website/generated-content/documentation/programming-guide/index.html
@@ -1844,16 +1844,16 @@ built-in transforms, you can <a href=/documentation/io/developing-io-overview/>i
 transforms</a>.</p><h3 id=pipeline-io-reading-data>5.1. Reading input data</h3><p>Read
transforms read data from an external source and return a <code>PCollection</code>
 representation of the data for use by your pipeline. You can use a read
 transform at any point while constructing your pipeline to create a new
-<code>PCollection</code>, though it will be most common at the start of your
pipeline.</p><div class="language-java snippet"><div class="notebook-skip code-snippet"><a
class=copy type=button data-bs-toggle=tooltip data-bs-placement=bottom title="Copy to clipboard"><img
src=/images/copy-icon.svg></a><div class=highlight><pre class=chroma><code
class=language-java data-lang=java><span class=n>PCollection</span><span
class=o>&lt;</span><span class=n>String</span><span class=o>&gt;</span>
<s [...]
+<code>PCollection</code>, though it will be most common at the start of your
pipeline.</p><div class="language-java snippet"><div class="notebook-skip code-snippet"><a
class=copy type=button data-bs-toggle=tooltip data-bs-placement=bottom title="Copy to clipboard"><img
src=/images/copy-icon.svg></a><div class=highlight><pre class=chroma><code
class=language-java data-lang=java><span class=n>PCollection</span><span
class=o>&lt;</span><span class=n>String</span><span class=o>&gt;</span>
<s [...]
 You will most often use write transforms at the end of your pipeline to output
 your pipeline&rsquo;s final results. However, you can use a write transform to output
-a <code>PCollection</code>'s data at any point in your pipeline.</p><div
class="language-java snippet"><div class="notebook-skip code-snippet"><a class=copy
type=button data-bs-toggle=tooltip data-bs-placement=bottom title="Copy to clipboard"><img
src=/images/copy-icon.svg></a><div class=highlight><pre class=chroma><code
class=language-java data-lang=java><span class=n>output</span><span class=o>.</span><span
class=na>apply</span><span class=o>(</span><span class=n>TextIO</span><span
cla [...]
+a <code>PCollection</code>'s data at any point in your pipeline.</p><div
class="language-java snippet"><div class="notebook-skip code-snippet"><a class=copy
type=button data-bs-toggle=tooltip data-bs-placement=bottom title="Copy to clipboard"><img
src=/images/copy-icon.svg></a><div class=highlight><pre class=chroma><code
class=language-java data-lang=java><span class=n>output</span><span class=o>.</span><span
class=na>apply</span><span class=o>(</span><span class=n>TextIO</span><span
cla [...]
 operator you provide. Note that glob operators are filesystem-specific and obey
 filesystem-specific consistency models. The following TextIO example uses a glob
 operator (<code>*</code>) to read all matching input files that have prefix &ldquo;input-&rdquo;
and the
 suffix &ldquo;.csv&rdquo; in the given location:</p><div class="language-java
snippet"><div class="notebook-skip code-snippet"><a class=copy type=button data-bs-toggle=tooltip
data-bs-placement=bottom title="Copy to clipboard"><img src=/images/copy-icon.svg></a><div
class=highlight><pre class=chroma><code class=language-java data-lang=java><span
class=n>p</span><span class=o>.</span><span class=na>apply</span><span
class=o>(</span><span class=s>&#34;ReadFromText&#34;</span><span
class=o> [...]
     <span class=n>TextIO</span><span class=o>.</span><span class=na>read</span><span
class=o>().</span><span class=na>from</span><span class=o>(</span><span
class=s>&#34;protocol://my_bucket/path/to/input-*.csv&#34;</span><span
class=o>));</span></code></pre></div></div></div><div
class="language-py snippet"><div class="notebook-skip code-snippet"><a class=copy
type=button data-bs-toggle=tooltip data-bs-placement=bottom title="Copy to clipboard"><img
src=/images/copy-icon.svg></a><div cl [...]
-    <span class=s1>&#39;path/to/input-*.csv&#39;</span><span class=p>)</span></code></pre></div></div></div><p>To
read data from disparate sources into a single <code>PCollection</code>, read
each one
+    <span class=s1>&#39;path/to/input-*.csv&#39;</span><span class=p>)</span></code></pre></div></div></div><div
class="language-go snippet"><div class="notebook-skip code-snippet"><a class=copy
type=button data-bs-toggle=tooltip data-bs-placement=bottom title="Copy to clipboard"><img
src=/images/copy-icon.svg></a><div class=highlight><pre class=chroma><code
class=language-go data-lang=go><span class=nx>lines</span> <span class=o>:=</span>
<span class=nx>textio</span><span class=p>.</spa [...]
 independently and then use the <a href=#flatten>Flatten</a> transform to create
a single
 <code>PCollection</code>.</p><h4 id=file-based-writing-multiple-files>5.3.2.
Writing to multiple output files</h4><p>For file-based output data, write transforms
write to multiple output files by
 default. When you pass an output file name to a write transform, the file name
@@ -1863,7 +1863,9 @@ location. Each file has the prefix &ldquo;numbers&rdquo;, a
numeric tag, and the
 &ldquo;.csv&rdquo;.</p><div class="language-java snippet"><div class="notebook-skip
code-snippet"><a class=copy type=button data-bs-toggle=tooltip data-bs-placement=bottom
title="Copy to clipboard"><img src=/images/copy-icon.svg></a><div class=highlight><pre
class=chroma><code class=language-java data-lang=java><span class=n>records</span><span
class=o>.</span><span class=na>apply</span><span class=o>(</span><span
class=s>&#34;WriteToText&#34;</span><span class=o>,</span>
     <span class=n>TextIO</span><span class=o>.</span><span class=na>write</span><span
class=o>().</span><span class=na>to</span><span class=o>(</span><span
class=s>&#34;protocol://my_bucket/path/to/numbers&#34;</span><span class=o>)</span>
                 <span class=o>.</span><span class=na>withSuffix</span><span
class=o>(</span><span class=s>&#34;.csv&#34;</span><span class=o>));</span></code></pre></div></div></div><div
class="language-py snippet"><div class="notebook-skip code-snippet"><a class=copy
type=button data-bs-toggle=tooltip data-bs-placement=bottom title="Copy to clipboard"><img
src=/images/copy-icon.svg></a><div class=highlight><pre class=chroma><code
class=language-py data-lang=py><span class=n>filtered_wor [...]
-    <span class=s1>&#39;/path/to/numbers&#39;</span><span class=p>,</span>
<span class=n>file_name_suffix</span><span class=o>=</span><span
class=s1>&#39;.csv&#39;</span><span class=p>)</span></code></pre></div></div></div><h3
id=provided-io-transforms>5.4. Beam-provided I/O transforms</h3><p>See the
<a href=/documentation/io/built-in/>Beam-provided I/O Transforms</a>
+    <span class=s1>&#39;/path/to/numbers&#39;</span><span class=p>,</span>
<span class=n>file_name_suffix</span><span class=o>=</span><span
class=s1>&#39;.csv&#39;</span><span class=p>)</span></code></pre></div></div></div><div
class="language-go snippet"><div class="notebook-skip code-snippet"><a class=copy
type=button data-bs-toggle=tooltip data-bs-placement=bottom title="Copy to clipboard"><img
src=/images/copy-icon.svg></a><div class=highlight><pre class=chroma><code
class=language-g [...]
+</span><span class=c1>// See https://issues.apache.org/jira/browse/BEAM-12664
for ways
+</span><span class=c1></span><span class=o>//</span> <span
class=nx>to</span> <span class=nx>contribute</span> <span class=nx>a</span>
<span class=nx>solution</span><span class=p>.</span></code></pre></div></div></div><h3
id=provided-io-transforms>5.4. Beam-provided I/O transforms</h3><p>See the
<a href=/documentation/io/built-in/>Beam-provided I/O Transforms</a>
 page for a list of the currently available I/O transforms.</p><h2 id=schemas>6.
Schemas</h2><p>Often, the types of the records being processed have an obvious
structure. Common Beam sources produce
 JSON, Avro, Protocol Buffer, or database row objects; all of these types have well defined
structures,
 structures that can often be determined by examining the type. Even within a SDK pipeline,
Simple Java POJOs
@@ -3919,7 +3921,7 @@ kafka_records = (
             ImplicitSchemaPayloadBuilder({&#39;data&#39;: u&#39;0&#39;}),
             &lt;Address of expansion service&gt;))
     assert_that(res, equal_to([&#39;0a&#39;, &#39;0b&#39;]))
-    </code></pre></div></div></li><li><p>After
the job has been submitted to the Beam runner, shutdown the expansion service by terminating
the expansion service process.</p></li></ol><h3 id=x-lang-transform-runner-support>13.3.
Runner Support</h3><p>Currently, portable runners such as Flink, Spark, and the
Direct runner can be used with multi-language pipelines.</p><p>Google Cloud Dataflow
supports multi-language pipelines through the Dataflow Runner v2 backend architecture.</p><div
cla [...]
+    </code></pre></div></div></li><li><p>After
the job has been submitted to the Beam runner, shutdown the expansion service by terminating
the expansion service process.</p></li></ol><h3 id=x-lang-transform-runner-support>13.3.
Runner Support</h3><p>Currently, portable runners such as Flink, Spark, and the
Direct runner can be used with multi-language pipelines.</p><p>Google Cloud Dataflow
supports multi-language pipelines through the Dataflow Runner v2 backend architecture.</p><div
cla [...]
 <a href=http://www.apache.org>The Apache Software Foundation</a>
 | <a href=/privacy_policy>Privacy Policy</a>
 | <a href=/feed.xml>RSS Feed</a><br><br>Apache Beam, Apache, Beam,
the Beam logo, and the Apache feather logo are either registered trademarks or trademarks
of The Apache Software Foundation. All other products or name brands are trademarks of their
respective holders, including The Apache Software Foundation.</div></div></div></div></footer></body></html>
\ No newline at end of file
diff --git a/website/generated-content/sitemap.xml b/website/generated-content/sitemap.xml
index a52845a..f5d773e 100644
--- a/website/generated-content/sitemap.xml
+++ b/website/generated-content/sitemap.xml
@@ -1 +1 @@
-<?xml version="1.0" encoding="utf-8" standalone="yes"?><urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9"
xmlns:xhtml="http://www.w3.org/1999/xhtml"><url><loc>/blog/beam-2.31.0/</loc><lastmod>2021-06-22T18:45:24-07:00</lastmod></url><url><loc>/categories/blog/</loc><lastmod>2021-07-01T15:48:01-07:00</lastmod></url><url><loc>/blog/</loc><lastmod>2021-07-01T15:48:01-07:00</lastmod></url><url><loc>/categories/</loc><lastmod>2021-07-01T15:48:01-07:00</lastmod></url><url><loc>/blog/b
[...]
\ No newline at end of file
+<?xml version="1.0" encoding="utf-8" standalone="yes"?><urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9"
xmlns:xhtml="http://www.w3.org/1999/xhtml"><url><loc>/blog/beam-2.31.0/</loc><lastmod>2021-06-22T18:45:24-07:00</lastmod></url><url><loc>/categories/blog/</loc><lastmod>2021-07-01T15:48:01-07:00</lastmod></url><url><loc>/blog/</loc><lastmod>2021-07-01T15:48:01-07:00</lastmod></url><url><loc>/categories/</loc><lastmod>2021-07-01T15:48:01-07:00</lastmod></url><url><loc>/blog/b
[...]
\ No newline at end of file

Mime
View raw message