beam-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From al...@apache.org
Subject [2/3] beam-site git commit: Regenerate website
Date Wed, 10 May 2017 04:25:42 GMT
Regenerate website


Project: http://git-wip-us.apache.org/repos/asf/beam-site/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam-site/commit/aa01f116
Tree: http://git-wip-us.apache.org/repos/asf/beam-site/tree/aa01f116
Diff: http://git-wip-us.apache.org/repos/asf/beam-site/diff/aa01f116

Branch: refs/heads/asf-site
Commit: aa01f1161bbbfc0060ab76ebad6bfcee7826e963
Parents: 8ed3f24
Author: Ahmet Altay <altay@google.com>
Authored: Tue May 9 21:24:35 2017 -0700
Committer: Ahmet Altay <altay@google.com>
Committed: Tue May 9 21:24:35 2017 -0700

----------------------------------------------------------------------
 content/documentation/runners/flink/index.html  |   2 +-
 .../sdks/java-extensions/index.html             | 237 +++++++++++++++++++
 content/documentation/sdks/java/index.html      |   9 +
 3 files changed, 247 insertions(+), 1 deletion(-)
----------------------------------------------------------------------


http://git-wip-us.apache.org/repos/asf/beam-site/blob/aa01f116/content/documentation/runners/flink/index.html
----------------------------------------------------------------------
diff --git a/content/documentation/runners/flink/index.html b/content/documentation/runners/flink/index.html
index 64ae583..deb04c8 100644
--- a/content/documentation/runners/flink/index.html
+++ b/content/documentation/runners/flink/index.html
@@ -287,7 +287,7 @@
 </tr>
 </table>
 
-<p>See the reference documentation for the  <span class="language-java"><a
href="/documentation/sdks/javadoc/0.6.0/index.html?org/apache/beam/runners/flink/FlinkPipelineOptions.html">FlinkPipelineOptions</a></span><span
class="language-py"><a href="https://github.com/apache/beam/blob/master/sdks/python/apache_beam/utils/pipeline_options.py">PipelineOptions</a></span>
interface (and its subinterfaces) for the complete list of pipeline configuration options.</p>
+<p>See the reference documentation for the  <span class="language-java"><a
href="/documentation/sdks/javadoc/0.6.0/index.html?org/apache/beam/runners/flink/FlinkPipelineOptions.html">FlinkPipelineOptions</a></span><span
class="language-py"><a href="https://github.com/apache/beam/blob/master/sdks/python/apache_beam/options/pipeline_options.py">PipelineOptions</a></span>
interface (and its subinterfaces) for the complete list of pipeline configuration options.</p>
 
 <h2 id="additional-information-and-caveats">Additional information and caveats</h2>
 

http://git-wip-us.apache.org/repos/asf/beam-site/blob/aa01f116/content/documentation/sdks/java-extensions/index.html
----------------------------------------------------------------------
diff --git a/content/documentation/sdks/java-extensions/index.html b/content/documentation/sdks/java-extensions/index.html
new file mode 100644
index 0000000..ee93238
--- /dev/null
+++ b/content/documentation/sdks/java-extensions/index.html
@@ -0,0 +1,237 @@
+<!DOCTYPE html>
+<html lang="en">
+
+  <head>
+  <meta charset="utf-8">
+  <meta http-equiv="X-UA-Compatible" content="IE=edge">
+  <meta name="viewport" content="width=device-width, initial-scale=1">
+
+  <title>Beam Java SDK Extensions</title>
+  <meta name="description" content="Apache Beam is an open source, unified model and set
of language-specific SDKs for defining and executing data processing workflows, and also data
ingestion and integration flows, supporting Enterprise Integration Patterns (EIPs) and Domain
Specific Languages (DSLs). Dataflow pipelines simplify the mechanics of large-scale batch
and streaming data processing and can run on a number of runtimes like Apache Flink, Apache
Spark, and Google Cloud Dataflow (a cloud service). Beam also brings DSL in different languages,
allowing users to easily implement their data integration processes.
+">
+
+  <link rel="stylesheet" href="/styles/site.css">
+  <link rel="stylesheet" href="/css/theme.css">
+  <script src="https://ajax.googleapis.com/ajax/libs/jquery/2.2.0/jquery.min.js"></script>
+  <script src="/js/bootstrap.min.js"></script>
+  <script src="/js/language-switch.js"></script>
+  <link rel="canonical" href="https://beam.apache.org/documentation/sdks/java-extensions/"
data-proofer-ignore>
+  <link rel="alternate" type="application/rss+xml" title="Apache Beam" href="https://beam.apache.org/feed.xml">
+  <script>
+    (function(i,s,o,g,r,a,m){i['GoogleAnalyticsObject']=r;i[r]=i[r]||function(){
+    (i[r].q=i[r].q||[]).push(arguments)},i[r].l=1*new Date();a=s.createElement(o),
+    m=s.getElementsByTagName(o)[0];a.async=1;a.src=g;m.parentNode.insertBefore(a,m)
+    })(window,document,'script','//www.google-analytics.com/analytics.js','ga');
+
+    ga('create', 'UA-73650088-1', 'auto');
+    ga('send', 'pageview');
+
+  </script>
+  <link rel="shortcut icon" type="image/x-icon" href="/images/favicon.ico">
+</head>
+
+
+  <body role="document">
+
+    <nav class="navbar navbar-default navbar-fixed-top">
+  <div class="container">
+    <div class="navbar-header">
+      <a href="/" class="navbar-brand" >
+        <img alt="Brand" style="height: 25px" src="/images/beam_logo_navbar.png">
+      </a>
+      <button type="button" class="navbar-toggle collapsed" data-toggle="collapse" data-target="#navbar"
aria-expanded="false" aria-controls="navbar">
+        <span class="sr-only">Toggle navigation</span>
+        <span class="icon-bar"></span>
+        <span class="icon-bar"></span>
+        <span class="icon-bar"></span>
+      </button>
+    </div>
+    <div id="navbar" class="navbar-collapse collapse">
+      <ul class="nav navbar-nav">
+        <li class="dropdown">
+		  <a href="#" class="dropdown-toggle" data-toggle="dropdown" role="button" aria-haspopup="true"
aria-expanded="false">Get Started <span class="caret"></span></a>
+		  <ul class="dropdown-menu">
+			  <li><a href="/get-started/beam-overview/">Beam Overview</a></li>
+        <li><a href="/get-started/quickstart-java/">Quickstart - Java</a></li>
+        <li><a href="/get-started/quickstart-py/">Quickstart - Python</a></li>
+			  <li role="separator" class="divider"></li>
+			  <li class="dropdown-header">Example Walkthroughs</li>
+			  <li><a href="/get-started/wordcount-example/">WordCount</a></li>
+			  <li><a href="/get-started/mobile-gaming-example/">Mobile Gaming</a></li>
+              <li role="separator" class="divider"></li>
+              <li class="dropdown-header">Resources</li>
+              <li><a href="/get-started/downloads">Downloads</a></li>
+              <li><a href="/get-started/support">Support</a></li>
+		  </ul>
+	    </li>
+        <li class="dropdown">
+		  <a href="#" class="dropdown-toggle" data-toggle="dropdown" role="button" aria-haspopup="true"
aria-expanded="false">Documentation <span class="caret"></span></a>
+		  <ul class="dropdown-menu">
+			  <li><a href="/documentation">Using the Documentation</a></li>
+			  <li role="separator" class="divider"></li>
+			  <li class="dropdown-header">Beam Concepts</li>
+			  <li><a href="/documentation/programming-guide/">Programming Guide</a></li>
+			  <li><a href="/documentation/resources/">Additional Resources</a></li>
+			  <li role="separator" class="divider"></li>
+              <li class="dropdown-header">Pipeline Fundamentals</li>
+              <li><a href="/documentation/pipelines/design-your-pipeline/">Design
Your Pipeline</a></li>
+              <li><a href="/documentation/pipelines/create-your-pipeline/">Create
Your Pipeline</a></li>
+              <li><a href="/documentation/pipelines/test-your-pipeline/">Test
Your Pipeline</a></li>
+              <li><a href="/documentation/io/io-toc/">Pipeline I/O</a></li>
+              <li role="separator" class="divider"></li>
+			  <li class="dropdown-header">SDKs</li>
+			  <li><a href="/documentation/sdks/java/">Java SDK</a></li>
+			  <li><a href="/documentation/sdks/javadoc/0.6.0/" target="_blank">Java SDK
API Reference <img src="/images/external-link-icon.png"
+                 width="14" height="14"
+                 alt="External link."></a>
+        </li>
+        <li><a href="/documentation/sdks/python/">Python SDK</a></li>
+        <li><a href="/documentation/sdks/pydoc/0.6.0/" target="_blank">Python
SDK API Reference <img src="/images/external-link-icon.png"
+                 width="14" height="14"
+                 alt="External link."></a>
+        </li>
+			  <li role="separator" class="divider"></li>
+			  <li class="dropdown-header">Runners</li>
+			  <li><a href="/documentation/runners/capability-matrix/">Capability Matrix</a></li>
+			  <li><a href="/documentation/runners/direct/">Direct Runner</a></li>
+			  <li><a href="/documentation/runners/apex/">Apache Apex Runner</a></li>
+			  <li><a href="/documentation/runners/flink/">Apache Flink Runner</a></li>
+			  <li><a href="/documentation/runners/spark/">Apache Spark Runner</a></li>
+			  <li><a href="/documentation/runners/dataflow/">Cloud Dataflow Runner</a></li>
+		  </ul>
+	    </li>
+        <li class="dropdown">
+		  <a href="#" class="dropdown-toggle" data-toggle="dropdown" role="button" aria-haspopup="true"
aria-expanded="false">Contribute <span class="caret"></span></a>
+		  <ul class="dropdown-menu">
+			  <li><a href="/contribute">Get Started Contributing</a></li>
+        <li role="separator" class="divider"></li>
+        <li class="dropdown-header">Guides</li>
+			  <li><a href="/contribute/contribution-guide/">Contribution Guide</a></li>
+        <li><a href="/contribute/testing/">Testing Guide</a></li>
+        <li><a href="/contribute/release-guide/">Release Guide</a></li>
+        <li><a href="/contribute/ptransform-style-guide/">PTransform Style Guide</a></li>
+        <li role="separator" class="divider"></li>
+        <li class="dropdown-header">Technical References</li>
+        <li><a href="/contribute/design-principles/">Design Principles</a></li>
+			  <li><a href="/contribute/work-in-progress/">Ongoing Projects</a></li>
+        <li><a href="/contribute/source-repository/">Source Repository</a></li>
+        <li role="separator" class="divider"></li>
+			  <li class="dropdown-header">Promotion</li>
+        <li><a href="/contribute/presentation-materials/">Presentation Materials</a></li>
+        <li><a href="/contribute/logos/">Logos and Design</a></li>
+        <li role="separator" class="divider"></li>
+        <li><a href="/contribute/maturity-model/">Maturity Model</a></li>
+        <li><a href="/contribute/team/">Team</a></li>
+		  </ul>
+	    </li>
+
+        <li><a href="/blog">Blog</a></li>
+      </ul>
+      <ul class="nav navbar-nav navbar-right">
+        <li class="dropdown">
+          <a href="#" class="dropdown-toggle" data-toggle="dropdown" role="button" aria-haspopup="true"
aria-expanded="false"><img src="https://www.apache.org/foundation/press/kit/feather_small.png"
alt="Apache Logo" style="height:24px;">Apache Software Foundation<span class="caret"></span></a>
+          <ul class="dropdown-menu dropdown-menu-right">
+            <li><a href="http://www.apache.org/">ASF Homepage</a></li>
+            <li><a href="http://www.apache.org/licenses/">License</a></li>
+            <li><a href="http://www.apache.org/security/">Security</a></li>
+            <li><a href="http://www.apache.org/foundation/thanks.html">Thanks</a></li>
+            <li><a href="http://www.apache.org/foundation/sponsorship.html">Sponsorship</a></li>
+            <li><a href="https://www.apache.org/foundation/policies/conduct">Code
of Conduct</a></li>
+          </ul>
+        </li>
+      </ul>
+    </div><!--/.nav-collapse -->
+  </div>
+</nav>
+
+
+<link rel="stylesheet" href="">
+
+
+    <div class="container" role="main">
+
+      <div class="row">
+        <h1 id="apache-beam-java-sdk-extensions">Apache Beam Java SDK Extensions</h1>
+
+<h2 id="a-namejoin-libraryajoin-library"><a name="join-library"></a>Join-library</h2>
+
+<p>Join-library provides inner join, outer left join, and outer right join functions.
The aim
+is to simplify the most common cases of join to a simple function call.</p>
+
+<p>The functions are generic and support joins of any Beam-supported types.
+Input to the join functions are <code class="highlighter-rouge">PCollections</code>
of <code class="highlighter-rouge">Key</code> / <code class="highlighter-rouge">Value</code>s.
Both
+the left and right <code class="highlighter-rouge">PCollection</code>s need the
same type for the key. All the join
+functions return a <code class="highlighter-rouge">Key</code> / <code class="highlighter-rouge">Value</code>
where <code class="highlighter-rouge">Key</code> is the join key and value is
+a <code class="highlighter-rouge">Key</code> / <code class="highlighter-rouge">Value</code>
where the key is the left value and right is the value.</p>
+
+<p>For outer joins, the user must provide a value that represents <code class="highlighter-rouge">null</code>
because <code class="highlighter-rouge">null</code>
+cannot be serialized.</p>
+
+<p>Example usage:</p>
+
+<div class="highlighter-rouge"><pre class="highlight"><code>PCollection&lt;KV&lt;String,
String&gt;&gt; leftPcollection = ...
+PCollection&lt;KV&lt;String, Long&gt;&gt; rightPcollection = ...
+
+PCollection&lt;KV&lt;String, KV&lt;String, Long&gt;&gt;&gt; joinedPcollection
=
+  Join.innerJoin(leftPcollection, rightPcollection);
+</code></pre>
+</div>
+
+<h2 id="a-namesorterasorter"><a name="sorter"></a>Sorter</h2>
+
+<p>This module provides the <code class="highlighter-rouge">SortValues</code>
transform, which takes a <code class="highlighter-rouge">PCollection&lt;KV&lt;K,
Iterable&lt;KV&lt;K2, V&gt;&gt;&gt;&gt;</code> and produces
a <code class="highlighter-rouge">PCollection&lt;KV&lt;K, Iterable&lt;KV&lt;K2,
V&gt;&gt;&gt;&gt;</code> where, for each primary key <code class="highlighter-rouge">K</code>
the paired <code class="highlighter-rouge">Iterable&lt;KV&lt;K2, V&gt;&gt;</code>
has been sorted by the byte encoding of secondary key (<code class="highlighter-rouge">K2</code>).
It is an efficient and scalable sorter for iterables, even if they are large (do not fit in
memory).</p>
+
+<h3 id="caveats">Caveats</h3>
+
+<ul>
+  <li>This transform performs value-only sorting; the iterable accompanying each key
is sorted, but <em>there is no relationship between different keys</em>, as Beam
does not support any defined relationship between different elements in a <code class="highlighter-rouge">PCollection</code>.</li>
+  <li>Each <code class="highlighter-rouge">Iterable&lt;KV&lt;K2, V&gt;&gt;</code>
is sorted on a single worker using local memory and disk. This means that <code class="highlighter-rouge">SortValues</code>
may be a performance and/or scalability bottleneck when used in different pipelines. For example,
users are discouraged from using <code class="highlighter-rouge">SortValues</code>
on a <code class="highlighter-rouge">PCollection</code> of a single element to
globally sort a large <code class="highlighter-rouge">PCollection</code>. A (rough)
estimate of the number of bytes of disk space utilized if sorting spills to disk is <code
class="highlighter-rouge">numRecords * (numSecondaryKeyBytesPerRecord + numValueBytesPerRecord
+ 16) * 3</code>.</li>
+</ul>
+
+<h3 id="options">Options</h3>
+
+<ul>
+  <li>The user can customize the temporary location used if sorting requires spilling
to disk and the maximum amount of memory to use by creating a custom instance of <code
class="highlighter-rouge">BufferedExternalSorter.Options</code> to pass into <code
class="highlighter-rouge">SortValues.create</code>.</li>
+</ul>
+
+<h3 id="example-usage-of-sortvalues">Example usage of <code class="highlighter-rouge">SortValues</code></h3>
+
+<div class="highlighter-rouge"><pre class="highlight"><code>PCollection&lt;KV&lt;String,
KV&lt;String, Integer&gt;&gt;&gt; input = ...
+
+// Group by primary key, bringing &lt;SecondaryKey, Value&gt; pairs for the same
key together.
+PCollection&lt;KV&lt;String, Iterable&lt;KV&lt;String, Integer&gt;&gt;&gt;&gt;
grouped =
+    input.apply(GroupByKey.&lt;String, KV&lt;String, Integer&gt;&gt;create());
+
+// For every primary key, sort the iterable of &lt;SecondaryKey, Value&gt; pairs
by secondary key.
+PCollection&lt;KV&lt;String, Iterable&lt;KV&lt;String, Integer&gt;&gt;&gt;&gt;
groupedAndSorted =
+    grouped.apply(
+        SortValues.&lt;String, String, Integer&gt;create(new BufferedExternalSorter.Options()));
+</code></pre>
+</div>
+
+      </div>
+
+
+    <hr>
+  <div class="row">
+      <div class="col-xs-12">
+          <footer>
+              <p class="text-center">
+                &copy; Copyright
+                <a href="http://www.apache.org">The Apache Software Foundation</a>,
+                2017. All Rights Reserved.
+              </p>
+              <p class="text-center">
+                <a href="/privacy_policy">Privacy Policy</a> |
+                <a href="/feed.xml">RSS Feed</a>
+              </p>
+          </footer>
+      </div>
+  </div>
+  <!-- container div end -->
+</div>
+
+
+  </body>
+
+</html>

http://git-wip-us.apache.org/repos/asf/beam-site/blob/aa01f116/content/documentation/sdks/java/index.html
----------------------------------------------------------------------
diff --git a/content/documentation/sdks/java/index.html b/content/documentation/sdks/java/index.html
index d44c20f..08d5819 100644
--- a/content/documentation/sdks/java/index.html
+++ b/content/documentation/sdks/java/index.html
@@ -168,6 +168,15 @@
 <h2 id="pipeline-io">Pipeline I/O</h2>
 <p>See the <a href="/documentation/io/built-in/">Beam-provided I/O Transforms</a>
page for a list of the currently available I/O transforms.</p>
 
+<h2 id="extensions">Extensions</h2>
+
+<p>The Java SDK has the following extensions:</p>
+
+<ul>
+  <li><a href="/documentation/sdks/java-extensions/#join-library">join-library</a>
provides inner join, outer left join, and outer right join functions.</li>
+  <li><a href="/documentation/sdks/java-extensions/#sorter">sorter</a>
is an efficient and scalable sorter for large iterables.</li>
+</ul>
+
       </div>
 
 


Mime
View raw message