Return-Path: X-Original-To: archive-asf-public-internal@cust-asf2.ponee.io Delivered-To: archive-asf-public-internal@cust-asf2.ponee.io Received: from cust-asf.ponee.io (cust-asf.ponee.io [163.172.22.183]) by cust-asf2.ponee.io (Postfix) with ESMTP id A9A23200BB7 for ; Wed, 26 Oct 2016 06:29:10 +0200 (CEST) Received: by cust-asf.ponee.io (Postfix) id A84E1160AFA; Wed, 26 Oct 2016 04:29:10 +0000 (UTC) Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by cust-asf.ponee.io (Postfix) with SMTP id 7B12A160AD8 for ; Wed, 26 Oct 2016 06:29:08 +0200 (CEST) Received: (qmail 12148 invoked by uid 500); 26 Oct 2016 04:29:07 -0000 Mailing-List: contact commits-help@beam.incubator.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@beam.incubator.apache.org Delivered-To: mailing list commits@beam.incubator.apache.org Received: (qmail 12139 invoked by uid 99); 26 Oct 2016 04:29:07 -0000 Received: from pnap-us-west-generic-nat.apache.org (HELO spamd1-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 26 Oct 2016 04:29:07 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd1-us-west.apache.org (ASF Mail Server at spamd1-us-west.apache.org) with ESMTP id E6671C0C51 for ; Wed, 26 Oct 2016 04:29:06 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd1-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: -6.219 X-Spam-Level: X-Spam-Status: No, score=-6.219 tagged_above=-999 required=6.31 tests=[KAM_ASCII_DIVIDERS=0.8, KAM_LAZY_DOMAIN_SECURITY=1, RCVD_IN_DNSWL_HI=-5, RCVD_IN_MSPIKE_H3=-0.01, RCVD_IN_MSPIKE_WL=-0.01, RP_MATCHES_RCVD=-2.999] autolearn=disabled Received: from mx1-lw-eu.apache.org ([10.40.0.8]) by localhost (spamd1-us-west.apache.org [10.40.0.7]) (amavisd-new, port 10024) with ESMTP id tvp-TBJReQOa for ; Wed, 26 Oct 2016 04:29:01 +0000 (UTC) Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by mx1-lw-eu.apache.org (ASF Mail Server at mx1-lw-eu.apache.org) with SMTP id 7B6A95FADA for ; Wed, 26 Oct 2016 04:28:59 +0000 (UTC) Received: (qmail 12007 invoked by uid 99); 26 Oct 2016 04:28:58 -0000 Received: from git1-us-west.apache.org (HELO git1-us-west.apache.org) (140.211.11.23) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 26 Oct 2016 04:28:58 +0000 Received: by git1-us-west.apache.org (ASF Mail Server at git1-us-west.apache.org, from userid 33) id 5A7FCE058E; Wed, 26 Oct 2016 04:28:58 +0000 (UTC) Content-Type: text/plain; charset="us-ascii" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit From: davor@apache.org To: commits@beam.incubator.apache.org Date: Wed, 26 Oct 2016 04:28:58 -0000 Message-Id: <8134498db52f4837a40145ed568c4190@git.apache.org> X-Mailer: ASF-Git Admin Mailer Subject: [1/2] incubator-beam-site git commit: Rebuild the site following merge of PR #50 archived-at: Wed, 26 Oct 2016 04:29:10 -0000 Repository: incubator-beam-site Updated Branches: refs/heads/asf-site 03542e8b7 -> 2302aee62 http://git-wip-us.apache.org/repos/asf/incubator-beam-site/blob/2302aee6/content/feed.xml ---------------------------------------------------------------------- diff --git a/content/feed.xml b/content/feed.xml index b36be59..9d5de98 100644 --- a/content/feed.xml +++ b/content/feed.xml @@ -6,8 +6,8 @@ http://beam.incubator.apache.org/ - Thu, 20 Oct 2016 17:09:19 -0700 - Thu, 20 Oct 2016 17:09:19 -0700 + Tue, 25 Oct 2016 21:24:51 -0700 + Tue, 25 Oct 2016 21:24:51 -0700 Jekyll v3.2.0 @@ -115,17 +115,17 @@ associated pane.</p> watermark and provide the result PCollection as input to the CalculateTeamScores PTransform:</p> -<div class="highlighter-rouge"><pre class="highlight"><code>TestStream&lt;GameActionInfo&gt; infos = TestStream.create(AvroCoder.of(GameActionInfo.class)) - .addElements(new GameActionInfo("sky", "blue", 12, new Instant(0L)), -                 new GameActionInfo("navy", "blue", 3, new Instant(0L)), -                 new GameActionInfo("navy", "blue", 3, new Instant(0L).plus(Duration.standardMinutes(3)))) -    // Move the watermark past the end the end of the window - .advanceWatermarkTo(new Instant(0L).plus(TEAM_WINDOW_DURATION) -                                 .plus(Duration.standardMinutes(1))) - .advanceWatermarkToInfinity(); - -PCollection&lt;KV&lt;String, Integer&gt;&gt; teamScores = p.apply(createEvents) - .apply(new CalculateTeamScores(TEAM_WINDOW_DURATION, ALLOWED_LATENESS)); +<div class="language-java highlighter-rouge"><pre class="highlight"><code><span class="n">TestStream</span><span class="o">&lt;</span><span class="n">GameActionInfo</span><span class="o">&gt;</span> <span class="n">infos</span> <span class="o">=</span> <span class="n">TestStream</span><span class="o">.</span><span class="na">create</span><span class="o">(</span><span class="n">AvroCoder</span><span class="o">.</span><span class="na">of</span><span class="o">(</span><span class="n">GameActionInfo</span><span class="o">.</span><span class="na">class</span><span class=&q uot;o">))</span> + <span class="o">.</span><span class="na">addElements</span><span class="o">(</span><span class="k">new</span> <span class="n">GameActionInfo</span><span class="o">(</span><span class="s">"sky"</span><span class="o">,</span> <span class="s">"blue"</span><span class="o">,</span> <span class="mi">12</span><span class="o">,</span> <span class="k">new</span> <span class="n">Instant</span><span class="o">(</span><span class="mi">0L</span><span class="o">)),</span> + <span class="err">                </span><span class="k">new</span> <span class="n">GameActionInfo</span><span class="o">(</span><span class="s">"navy"</span><span class="o">,</span> <span class="s">"blue"</span><span class="o">,</span> <span class="mi">3</span><span class="o">,</span> <span class="k">new</span> <span class="n">Instant</span><span class="o">(</span><span class="mi">0L</span><span class="o">)),</span> + <span class="err">                </span><span class="k">new</span> <span class="n">GameActionInfo</span><span class="o">(</span><span class="s">"navy"</span><span class="o">,</span> <span class="s">"blue"</span><span class="o">,</span> <span class="mi">3</span><span class="o">,</span> <span class="k">new</span> <span class="n">Instant</span><span class="o">(</span><span class="mi">0L</span><span class="o">).</span><span class="na">plus</span><span class="o">(</span><span class="n">Duration</span><span class="o">.</span><span class=& quot;na">standardMinutes</span><span class="o">(</span><span class="mi">3</span><span class="o">))))</span> + <span class="err">   </span><span class="c1">// Move the watermark past the end the end of the window</span> + <span class="o">.</span><span class="na">advanceWatermarkTo</span><span class="o">(</span><span class="k">new</span> <span class="n">Instant</span><span class="o">(</span><span class="mi">0L</span><span class="o">).</span><span class="na">plus</span><span class="o">(</span><span class="n">TEAM_WINDOW_DURATION</span><span class="o">)</span> + <span class="err">                               </span> <span class="o">.</span><span class="na">plus</span><span class="o">(</span><span class="n">Duration</span><span class="o">.</span><span class="na">standardMinutes</span><span class="o">(</span><span class="mi">1</span><span class="o">)))</span> + <span class="o">.</span><span class="na">advanceWatermarkToInfinity</span><span class="o">();</span> + +<span class="n">PCollection</span><span class="o">&lt;</span><span class="n">KV</span><span class="o">&lt;</span><span class="n">String</span><span class="o">,</span> <span class="n">Integer</span><span class="o">&gt;&gt;</span> <span class="n">teamScores</span> <span class="o">=</span> <span class="n">p</span><span class="o">.</span><span class="na">apply</span><span class="o">(</span><span class="n">createEvents</span><span class="o">)</span> + <span class="o">.</span><span class="na">apply</span><span class="o">(</span><span class="k">new</span> <span class="n">CalculateTeamScores</span><span class="o">(</span><span class="n">TEAM_WINDOW_DURATION</span><span class="o">,</span> <span class="n">ALLOWED_LATENESS</span><span class="o">));</span> </code></pre> </div> @@ -133,11 +133,11 @@ PCollection&lt;KV&lt;String, Integer&gt;&gt; teamScores = p.appl <p><img class="center-block" src="/images/blog/test-stream/elements-all-on-time.png" alt="Elements all arrive before the watermark, and are produced in the on-time pane" width="442" /></p> -<div class="highlighter-rouge"><pre class="highlight"><code>// Only one value is emitted for the blue team -PAssert.that(teamScores) - .inWindow(window) - .containsInAnyOrder(KV.of("blue", 18)); -p.run(); +<div class="language-java highlighter-rouge"><pre class="highlight"><code><span class="c1">// Only one value is emitted for the blue team</span> +<span class="n">PAssert</span><span class="o">.</span><span class="na">that</span><span class="o">(</span><span class="n">teamScores</span><span class="o">)</span> + <span class="o">.</span><span class="na">inWindow</span><span class="o">(</span><span class="n">window</span><span class="o">)</span> + <span class="o">.</span><span class="na">containsInAnyOrder</span><span class="o">(</span><span class="n">KV</span><span class="o">.</span><span class="na">of</span><span class="o">(</span><span class="s">"blue"</span><span class="o">,</span> <span class="mi">18</span><span class="o">));</span> +<span class="n">p</span><span class="o">.</span><span class="na">run</span><span class="o">();</span> </code></pre> </div> @@ -149,27 +149,27 @@ of the window (shown below to the left of the red watermark), which demonstrates the system to be on time, as it arrives before the watermark passes the end of the window</p> -<div class="highlighter-rouge"><pre class="highlight"><code>TestStream&lt;GameActionInfo&gt; infos = TestStream.create(AvroCoder.of(GameActionInfo.class)) - .addElements(new GameActionInfo("sky", "blue", 3, new Instant(0L)), -         new GameActionInfo("navy", "blue", 3, new Instant(0L).plus(Duration.standardMinutes(3)))) -    // Move the watermark up to "near" the end of the window - .advanceWatermarkTo(new Instant(0L).plus(TEAM_WINDOW_DURATION) -                                 .minus(Duration.standardMinutes(1))) - .addElements(new GameActionInfo("sky", "blue", 12, Duration.ZERO)) - .advanceWatermarkToInfinity(); - -PCollection&lt;KV&lt;String, Integer&gt;&gt; teamScores = p.apply(createEvents) - .apply(new CalculateTeamScores(TEAM_WINDOW_DURATION, ALLOWED_LATENESS)); +<div class="language-java highlighter-rouge"><pre class="highlight"><code><span class="n">TestStream</span><span class="o">&lt;</span><span class="n">GameActionInfo</span><span class="o">&gt;</span> <span class="n">infos</span> <span class="o">=</span> <span class="n">TestStream</span><span class="o">.</span><span class="na">create</span><span class="o">(</span><span class="n">AvroCoder</span><span class="o">.</span><span class="na">of</span><span class="o">(</span><span class="n">GameActionInfo</span><span class="o">.</span><span class="na">class</span><span class=&q uot;o">))</span> + <span class="o">.</span><span class="na">addElements</span><span class="o">(</span><span class="k">new</span> <span class="n">GameActionInfo</span><span class="o">(</span><span class="s">"sky"</span><span class="o">,</span> <span class="s">"blue"</span><span class="o">,</span> <span class="mi">3</span><span class="o">,</span> <span class="k">new</span> <span class="n">Instant</span><span class="o">(</span><span class="mi">0L</span><span class="o">)),</span> + <span class="err">       </span> <span class="k">new</span> <span class="n">GameActionInfo</span><span class="o">(</span><span class="s">"navy"</span><span class="o">,</span> <span class="s">"blue"</span><span class="o">,</span> <span class="mi">3</span><span class="o">,</span> <span class="k">new</span> <span class="n">Instant</span><span class="o">(</span><span class="mi">0L</span><span class="o">).</span><span class="na">plus</span><span class="o">(</span><span class="n">Duration</span><span class="o">.</span><span class="na&q uot;>standardMinutes</span><span class="o">(</span><span class="mi">3</span><span class="o">))))</span> + <span class="err">   </span><span class="c1">// Move the watermark up to "near" the end of the window</span> + <span class="o">.</span><span class="na">advanceWatermarkTo</span><span class="o">(</span><span class="k">new</span> <span class="n">Instant</span><span class="o">(</span><span class="mi">0L</span><span class="o">).</span><span class="na">plus</span><span class="o">(</span><span class="n">TEAM_WINDOW_DURATION</span><span class="o">)</span> + <span class="err">               </span> <span class="err">                </span><span class="o">.</span><span class="na">minus</span><span class="o">(</span><span class="n">Duration</span><span class="o">.</span><span class="na">standardMinutes</span><span class="o">(</span><span class="mi">1</span><span class="o">)))</span> + <span class="o">.</span><span class="na">addElements</span><span class="o">(</span><span class="k">new</span> <span class="n">GameActionInfo</span><span class="o">(</span><span class="s">"sky"</span><span class="o">,</span> <span class="s">"blue"</span><span class="o">,</span> <span class="mi">12</span><span class="o">,</span> <span class="n">Duration</span><span class="o">.</span><span class="na">ZERO</span><span class="o">))</span> + <span class="o">.</span><span class="na">advanceWatermarkToInfinity</span><span class="o">();</span> + +<span class="n">PCollection</span><span class="o">&lt;</span><span class="n">KV</span><span class="o">&lt;</span><span class="n">String</span><span class="o">,</span> <span class="n">Integer</span><span class="o">&gt;&gt;</span> <span class="n">teamScores</span> <span class="o">=</span> <span class="n">p</span><span class="o">.</span><span class="na">apply</span><span class="o">(</span><span class="n">createEvents</span><span class="o">)</span> + <span class="o">.</span><span class="na">apply</span><span class="o">(</span><span class="k">new</span> <span class="n">CalculateTeamScores</span><span class="o">(</span><span class="n">TEAM_WINDOW_DURATION</span><span class="o">,</span> <span class="n">ALLOWED_LATENESS</span><span class="o">));</span> </code></pre> </div> <p><img class="center-block" src="/images/blog/test-stream/elements-unobservably-late.png" alt="An element arrives late, but before the watermark passes the end of the window, and is produced in the on-time pane" width="442" /></p> -<div class="highlighter-rouge"><pre class="highlight"><code>// Only one value is emitted for the blue team -PAssert.that(teamScores) - .inWindow(window) - .containsInAnyOrder(KV.of("blue", 18)); -p.run(); +<div class="language-java highlighter-rouge"><pre class="highlight"><code><span class="c1">// Only one value is emitted for the blue team</span> +<span class="n">PAssert</span><span class="o">.</span><span class="na">that</span><span class="o">(</span><span class="n">teamScores</span><span class="o">)</span> + <span class="o">.</span><span class="na">inWindow</span><span class="o">(</span><span class="n">window</span><span class="o">)</span> + <span class="o">.</span><span class="na">containsInAnyOrder</span><span class="o">(</span><span class="n">KV</span><span class="o">.</span><span class="na">of</span><span class="o">(</span><span class="s">"blue"</span><span class="o">,</span> <span class="mi">18</span><span class="o">));</span> +<span class="n">p</span><span class="o">.</span><span class="na">run</span><span class="o">();</span> </code></pre> </div> @@ -179,31 +179,31 @@ p.run(); demonstrate the triggering behavior that causes the system to emit an on-time pane, and then after the late data arrives, a pane that refines the result.</p> -<div class="highlighter-rouge"><pre class="highlight"><code>TestStream&lt;GameActionInfo&gt; infos = TestStream.create(AvroCoder.of(GameActionInfo.class)) - .addElements(new GameActionInfo("sky", "blue", 3, new Instant(0L)), -           new GameActionInfo("navy", "blue", 3, new Instant(0L).plus(Duration.standardMinutes(3)))) -    // Move the watermark up to "near" the end of the window - .advanceWatermarkTo(new Instant(0L).plus(TEAM_WINDOW_DURATION) -                                 .minus(Duration.standardMinutes(1))) - .addElements(new GameActionInfo("sky", "blue", 12, Duration.ZERO)) - .advanceWatermarkToInfinity(); - -PCollection&lt;KV&lt;String, Integer&gt;&gt; teamScores = p.apply(createEvents) - .apply(new CalculateTeamScores(TEAM_WINDOW_DURATION, ALLOWED_LATENESS)); +<div class="language-java highlighter-rouge"><pre class="highlight"><code><span class="n">TestStream</span><span class="o">&lt;</span><span class="n">GameActionInfo</span><span class="o">&gt;</span> <span class="n">infos</span> <span class="o">=</span> <span class="n">TestStream</span><span class="o">.</span><span class="na">create</span><span class="o">(</span><span class="n">AvroCoder</span><span class="o">.</span><span class="na">of</span><span class="o">(</span><span class="n">GameActionInfo</span><span class="o">.</span><span class="na">class</span><span class=&q uot;o">))</span> + <span class="o">.</span><span class="na">addElements</span><span class="o">(</span><span class="k">new</span> <span class="n">GameActionInfo</span><span class="o">(</span><span class="s">"sky"</span><span class="o">,</span> <span class="s">"blue"</span><span class="o">,</span> <span class="mi">3</span><span class="o">,</span> <span class="k">new</span> <span class="n">Instant</span><span class="o">(</span><span class="mi">0L</span><span class="o">)),</span> + <span class="err">          </span><span class="k">new</span> <span class="n">GameActionInfo</span><span class="o">(</span><span class="s">"navy"</span><span class="o">,</span> <span class="s">"blue"</span><span class="o">,</span> <span class="mi">3</span><span class="o">,</span> <span class="k">new</span> <span class="n">Instant</span><span class="o">(</span><span class="mi">0L</span><span class="o">).</span><span class="na">plus</span><span class="o">(</span><span class="n">Duration</span><span class="o">.</span><span class="n a">standardMinutes</span><span class="o">(</span><span class="mi">3</span><span class="o">))))</span> +<span class="err"> </span> <span class="err">  </span><span class="c1">// Move the watermark up to "near" the end of the window</span> + <span class="o">.</span><span class="na">advanceWatermarkTo</span><span class="o">(</span><span class="k">new</span> <span class="n">Instant</span><span class="o">(</span><span class="mi">0L</span><span class="o">).</span><span class="na">plus</span><span class="o">(</span><span class="n">TEAM_WINDOW_DURATION</span><span class="o">)</span> +<span class="err">                                </span> <span class="o">.</span><span class="na">minus</span><span class="o">(</span><span class="n">Duration</span><span class="o">.</span><span class="na">standardMinutes</span><span class="o">(</span><span class="mi">1</span><span class="o">)))</span> + <span class="o">.</span><span class="na">addElements</span><span class="o">(</span><span class="k">new</span> <span class="n">GameActionInfo</span><span class="o">(</span><span class="s">"sky"</span><span class="o">,</span> <span class="s">"blue"</span><span class="o">,</span> <span class="mi">12</span><span class="o">,</span> <span class="n">Duration</span><span class="o">.</span><span class="na">ZERO</span><span class="o">))</span> + <span class="o">.</span><span class="na">advanceWatermarkToInfinity</span><span class="o">();</span> + +<span class="n">PCollection</span><span class="o">&lt;</span><span class="n">KV</span><span class="o">&lt;</span><span class="n">String</span><span class="o">,</span> <span class="n">Integer</span><span class="o">&gt;&gt;</span> <span class="n">teamScores</span> <span class="o">=</span> <span class="n">p</span><span class="o">.</span><span class="na">apply</span><span class="o">(</span><span class="n">createEvents</span><span class="o">)</span> + <span class="o">.</span><span class="na">apply</span><span class="o">(</span><span class="k">new</span> <span class="n">CalculateTeamScores</span><span class="o">(</span><span class="n">TEAM_WINDOW_DURATION</span><span class="o">,</span> <span class="n">ALLOWED_LATENESS</span><span class="o">));</span> </code></pre> </div> <p><img class="center-block" src="/images/blog/test-stream/elements-observably-late.png" alt="Elements all arrive before the watermark, and are produced in the on-time pane" width="442" /></p> -<div class="highlighter-rouge"><pre class="highlight"><code>// An on-time pane is emitted with the events that arrived before the window closed -PAssert.that(teamScores) - .inOnTimePane(window) - .containsInAnyOrder(KV.of("blue", 6)); -// The final pane contains the late refinement -PAssert.that(teamScores) - .inFinalPane(window) - .containsInAnyOrder(KV.of("blue", 18)); -p.run(); +<div class="language-java highlighter-rouge"><pre class="highlight"><code><span class="c1">// An on-time pane is emitted with the events that arrived before the window closed</span> +<span class="n">PAssert</span><span class="o">.</span><span class="na">that</span><span class="o">(</span><span class="n">teamScores</span><span class="o">)</span> + <span class="o">.</span><span class="na">inOnTimePane</span><span class="o">(</span><span class="n">window</span><span class="o">)</span> + <span class="o">.</span><span class="na">containsInAnyOrder</span><span class="o">(</span><span class="n">KV</span><span class="o">.</span><span class="na">of</span><span class="o">(</span><span class="s">"blue"</span><span class="o">,</span> <span class="mi">6</span><span class="o">));</span> +<span class="c1">// The final pane contains the late refinement</span> +<span class="n">PAssert</span><span class="o">.</span><span class="na">that</span><span class="o">(</span><span class="n">teamScores</span><span class="o">)</span> + <span class="o">.</span><span class="na">inFinalPane</span><span class="o">(</span><span class="n">window</span><span class="o">)</span> + <span class="o">.</span><span class="na">containsInAnyOrder</span><span class="o">(</span><span class="n">KV</span><span class="o">.</span><span class="na">of</span><span class="o">(</span><span class="s">"blue"</span><span class="o">,</span> <span class="mi">18</span><span class="o">));</span> +<span class="n">p</span><span class="o">.</span><span class="na">run</span><span class="o">();</span> </code></pre> </div> @@ -213,33 +213,33 @@ p.run(); configured allowed lateness, we can demonstrate that the late element is dropped by the system.</p> -<div class="highlighter-rouge"><pre class="highlight"><code>TestStream&lt;GameActionInfo&gt; infos = TestStream.create(AvroCoder.of(GameActionInfo.class)) - .addElements(new GameActionInfo("sky", "blue", 3, Duration.ZERO), -          new GameActionInfo("navy", "blue", 3, Duration.standardMinutes(3))) -    // Move the watermark up to "near" the end of the window - .advanceWatermarkTo(new Instant(0).plus(TEAM_WINDOW_DURATION) -                                         .plus(ALLOWED_LATENESS) -                                         .plus(Duration.standardMinutes(1))) - .addElements(new GameActionInfo( -                     "sky", -                     "blue", -                     12, -                     new Instant(0).plus(TEAM_WINDOW_DURATION).minus(Duration.standardMinutes(1)))) - .advanceWatermarkToInfinity(); - -PCollection&lt;KV&lt;String, Integer&gt;&gt; teamScores = p.apply(createEvents) - .apply(new CalculateTeamScores(TEAM_WINDOW_DURATION, ALLOWED_LATENESS)); +<div class="language-java highlighter-rouge"><pre class="highlight"><code><span class="n">TestStream</span><span class="o">&lt;</span><span class="n">GameActionInfo</span><span class="o">&gt;</span> <span class="n">infos</span> <span class="o">=</span> <span class="n">TestStream</span><span class="o">.</span><span class="na">create</span><span class="o">(</span><span class="n">AvroCoder</span><span class="o">.</span><span class="na">of</span><span class="o">(</span><span class="n">GameActionInfo</span><span class="o">.</span><span class="na">class</span><span class=&q uot;o">))</span> + <span class="o">.</span><span class="na">addElements</span><span class="o">(</span><span class="k">new</span> <span class="n">GameActionInfo</span><span class="o">(</span><span class="s">"sky"</span><span class="o">,</span> <span class="s">"blue"</span><span class="o">,</span> <span class="mi">3</span><span class="o">,</span> <span class="n">Duration</span><span class="o">.</span><span class="na">ZERO</span><span class="o">),</span> + <span class="err">         </span><span class="k">new</span> <span class="n">GameActionInfo</span><span class="o">(</span><span class="s">"navy"</span><span class="o">,</span> <span class="s">"blue"</span><span class="o">,</span> <span class="mi">3</span><span class="o">,</span> <span class="n">Duration</span><span class="o">.</span><span class="na">standardMinutes</span><span class="o">(</span><span class="mi">3</span><span class="o">)))</span> +<span class="err">    </span><span class="c1">// Move the watermark up to "near" the end of the window</span> + <span class="o">.</span><span class="na">advanceWatermarkTo</span><span class="o">(</span><span class="k">new</span> <span class="n">Instant</span><span class="o">(</span><span class="mi">0</span><span class="o">).</span><span class="na">plus</span><span class="o">(</span><span class="n">TEAM_WINDOW_DURATION</span><span class="o">)</span> + <span class="err">                                        </span><span class="o">.</span><span class="na">plus</span><span class="o">(</span><span class="n">ALLOWED_LATENESS</span><span class="o">)</span> + <span class="err">                                        </span><span class="o">.</span><span class="na">plus</span><span class="o">(</span><span class="n">Duration</span><span class="o">.</span><span class="na">standardMinutes</span><span class="o">(</span><span class="mi">1</span><span class="o">)))</span> + <span class="o">.</span><span class="na">addElements</span><span class="o">(</span><span class="k">new</span> <span class="n">GameActionInfo</span><span class="o">(</span> +<span class="err">                     </span><span class="s">"sky"</span><span class="o">,</span> +<span class="err">                     </span><span class="s">"blue"</span><span class="o">,</span> +<span class="err">                     </span><span class="mi">12</span><span class="o">,</span> +<span class="err">                     </span><span class="k">new</span> <span class="n">Instant</span><span class="o">(</span><span class="mi">0</span><span class="o">).</span><span class="na">plus</span><span class="o">(</span><span class="n">TEAM_WINDOW_DURATION</span><span class="o">).</span><span class="na">minus</span><span class="o">(</span><span class="n">Duration</span><span class="o">.</span><span class="na">standardMinutes</span><span class="o">(</span><span class="mi">1</span><span class="o">))))</span> + <span class="o">.</span><span class="na">advanceWatermarkToInfinity</span><span class="o">();</span> + +<span class="n">PCollection</span><span class="o">&lt;</span><span class="n">KV</span><span class="o">&lt;</span><span class="n">String</span><span class="o">,</span> <span class="n">Integer</span><span class="o">&gt;&gt;</span> <span class="n">teamScores</span> <span class="o">=</span> <span class="n">p</span><span class="o">.</span><span class="na">apply</span><span class="o">(</span><span class="n">createEvents</span><span class="o">)</span> + <span class="o">.</span><span class="na">apply</span><span class="o">(</span><span class="k">new</span> <span class="n">CalculateTeamScores</span><span class="o">(</span><span class="n">TEAM_WINDOW_DURATION</span><span class="o">,</span> <span class="n">ALLOWED_LATENESS</span><span class="o">));</span> </code></pre> </div> <p><img class="center-block" src="/images/blog/test-stream/elements-droppably-late.png" alt="Elements all arrive before the watermark, and are produced in the on-time pane" width="442" /></p> -<div class="highlighter-rouge"><pre class="highlight"><code>// An on-time pane is emitted with the events that arrived before the window closed -PAssert.that(teamScores) - .inWindow(window) - .containsInAnyOrder(KV.of("blue", 6)); +<div class="language-java highlighter-rouge"><pre class="highlight"><code><span class="c1">// An on-time pane is emitted with the events that arrived before the window closed</span> +<span class="n">PAssert</span><span class="o">.</span><span class="na">that</span><span class="o">(</span><span class="n">teamScores</span><span class="o">)</span> + <span class="o">.</span><span class="na">inWindow</span><span class="o">(</span><span class="n">window</span><span class="o">)</span> + <span class="o">.</span><span class="na">containsInAnyOrder</span><span class="o">(</span><span class="n">KV</span><span class="o">.</span><span class="na">of</span><span class="o">(</span><span class="s">"blue"</span><span class="o">,</span> <span class="mi">6</span><span class="o">));</span> -p.run(); +<span class="n">p</span><span class="o">.</span><span class="na">run</span><span class="o">();</span> </code></pre> </div> @@ -249,29 +249,29 @@ triggers by advancing the processing time of the TestStream. If we add elements to an input PCollection, occasionally advancing the processing time clock, and apply <code class="highlighter-rouge">CalculateUserScores</code></p> -<div class="highlighter-rouge"><pre class="highlight"><code>TestStream.create(AvroCoder.of(GameActionInfo.class)) -    .addElements(new GameActionInfo("scarlet", "red", 3, new Instant(0L)), -                new GameActionInfo("scarlet", "red", 2, new Instant(0L).plus(Duration.standardMinutes(1)))) - .advanceProcessingTime(Duration.standardMinutes(12)) -    .addElements(new GameActionInfo("oxblood", "red", 2, new Instant(0L)).plus(Duration.standardSeconds(22)), -                new GameActionInfo("scarlet", "red", 4, new Instant(0L).plus(Duration.standardMinutes(2)))) - .advanceProcessingTime(Duration.standardMinutes(15)) - .advanceWatermarkToInfinity(); - -PCollection&lt;KV&lt;String, Integer&gt;&gt; userScores = -    p.apply(infos).apply(new CalculateUserScores(ALLOWED_LATENESS)); +<div class="language-java highlighter-rouge"><pre class="highlight"><code><span class="n">TestStream</span><span class="o">.</span><span class="na">create</span><span class="o">(</span><span class="n">AvroCoder</span><span class="o">.</span><span class="na">of</span><span class="o">(</span><span class="n">GameActionInfo</span><span class="o">.</span><span class="na">class</span><span class="o">))</span> + <span class="err">   </span><span class="o">.</span><span class="na">addElements</span><span class="o">(</span><span class="k">new</span> <span class="n">GameActionInfo</span><span class="o">(</span><span class="s">"scarlet"</span><span class="o">,</span> <span class="s">"red"</span><span class="o">,</span> <span class="mi">3</span><span class="o">,</span> <span class="k">new</span> <span class="n">Instant</span><span class="o">(</span><span class="mi">0L</span><span class="o">)),</span> + <span class="err">              </span> <span class="k">new</span> <span class="n">GameActionInfo</span><span class="o">(</span><span class="s">"scarlet"</span><span class="o">,</span> <span class="s">"red"</span><span class="o">,</span> <span class="mi">2</span><span class="o">,</span> <span class="k">new</span> <span class="n">Instant</span><span class="o">(</span><span class="mi">0L</span><span class="o">).</span><span class="na">plus</span><span class="o">(</span><span class="n">Duration</span><span class="o">.</span><span class=& quot;na">standardMinutes</span><span class="o">(</span><span class="mi">1</span><span class="o">))))</span> + <span class="o">.</span><span class="na">advanceProcessingTime</span><span class="o">(</span><span class="n">Duration</span><span class="o">.</span><span class="na">standardMinutes</span><span class="o">(</span><span class="mi">12</span><span class="o">))</span> + <span class="err">   </span><span class="o">.</span><span class="na">addElements</span><span class="o">(</span><span class="k">new</span> <span class="n">GameActionInfo</span><span class="o">(</span><span class="s">"oxblood"</span><span class="o">,</span> <span class="s">"red"</span><span class="o">,</span> <span class="mi">2</span><span class="o">,</span> <span class="k">new</span> <span class="n">Instant</span><span class="o">(</span><span class="mi">0L</span><span class="o">)).</span><span class="na">plus</span><span class="o">(&l t;/span><span class="n">Duration</span><span class="o">.</span><span class="na">standardSeconds</span><span class="o">(</span><span class="mi">22</span><span class="o">)),</span> + <span class="err">              </span> <span class="k">new</span> <span class="n">GameActionInfo</span><span class="o">(</span><span class="s">"scarlet"</span><span class="o">,</span> <span class="s">"red"</span><span class="o">,</span> <span class="mi">4</span><span class="o">,</span> <span class="k">new</span> <span class="n">Instant</span><span class="o">(</span><span class="mi">0L</span><span class="o">).</span><span class="na">plus</span><span class="o">(</span><span class="n">Duration</span><span class="o">.</span><span class=& quot;na">standardMinutes</span><span class="o">(</span><span class="mi">2</span><span class="o">))))</span> + <span class="o">.</span><span class="na">advanceProcessingTime</span><span class="o">(</span><span class="n">Duration</span><span class="o">.</span><span class="na">standardMinutes</span><span class="o">(</span><span class="mi">15</span><span class="o">))</span> + <span class="o">.</span><span class="na">advanceWatermarkToInfinity</span><span class="o">();</span> + +<span class="n">PCollection</span><span class="o">&lt;</span><span class="n">KV</span><span class="o">&lt;</span><span class="n">String</span><span class="o">,</span> <span class="n">Integer</span><span class="o">&gt;&gt;</span> <span class="n">userScores</span> <span class="o">=</span> + <span class="err">   </span><span class="n">p</span><span class="o">.</span><span class="na">apply</span><span class="o">(</span><span class="n">infos</span><span class="o">).</span><span class="na">apply</span><span class="o">(</span><span class="k">new</span> <span class="n">CalculateUserScores</span><span class="o">(</span><span class="n">ALLOWED_LATENESS</span><span class="o">));</span> </code></pre> </div> <p><img class="center-block" src="/images/blog/test-stream/elements-processing-speculative.png" alt="Elements all arrive before the watermark, and are produced in the on-time pane" width="442" /></p> -<div class="highlighter-rouge"><pre class="highlight"><code>PAssert.that(userScores) - .inEarlyGlobalWindowPanes() - .containsInAnyOrder(KV.of("scarlet", 5), -                         KV.of("scarlet", 9), - KV.of("oxblood", 2)); +<div class="language-java highlighter-rouge"><pre class="highlight"><code><span class="n">PAssert</span><span class="o">.</span><span class="na">that</span><span class="o">(</span><span class="n">userScores</span><span class="o">)</span> + <span class="o">.</span><span class="na">inEarlyGlobalWindowPanes</span><span class="o">()</span> + <span class="o">.</span><span class="na">containsInAnyOrder</span><span class="o">(</span><span class="n">KV</span><span class="o">.</span><span class="na">of</span><span class="o">(</span><span class="s">"scarlet"</span><span class="o">,</span> <span class="mi">5</span><span class="o">),</span> + <span class="err">                        </span><span class="n">KV</span><span class="o">.</span><span class="na">of</span><span class="o">(</span><span class="s">"scarlet"</span><span class="o">,</span> <span class="mi">9</span><span class="o">),</span> + <span class="n">KV</span><span class="o">.</span><span class="na">of</span><span class="o">(</span><span class="s">"oxblood"</span><span class="o">,</span> <span class="mi">2</span><span class="o">));</span> -p.run(); +<span class="n">p</span><span class="o">.</span><span class="na">run</span><span class="o">();</span> </code></pre> </div> @@ -621,7 +621,7 @@ PCollection&lt;O&gt; output = input </li> </ul> -<p>As Apache Beam grows, so will this repository of presentation materials. We are excited to add new materials as the Apache Beam ecosystem grows with new runners, SDKs, and so on. If you are interested in contributing content or have a request, please see the <a href="/learn/presentation-materials/">Apache Beam presentation materials</a> page or email the <a href="&#109;&#097;&#105;&#108;&#116;&#111;:&#117;&#115;&#101;&#114;&#064;&#098;&#101;&#097;&#109;&#046;&#105;&#110;&#099;&#117;&#098;&#097;&#116;&#111;&#114;&#046;&#097;&#112;&#097;&#099;&#104;&#101;&#046;&#111;&#114;&#103;"><code class="highlighter-rouge">user@beam.incubator.apache.org</code></a> mailing list with your ideas or questions.</p> +<p>As Apache Beam grows, so will this repository of presentation materials. We are excited to add new materials as the Apache Beam ecosystem grows with new runners, SDKs, and so on. If you are interested in contributing content or have a request, please see the <a href="/learn/presentation-materials/">Apache Beam presentation materials</a> page or email the <a href="mailto:user@beam.incubator.apache.org"><code class="highlighter-rouge">user@beam.incubator.apache.org</code></a> mailing list with your ideas or questions.</p> Sun, 03 Apr 2016 11:00:00 -0700 http://beam.incubator.apache.org/beam/capability/2016/04/03/presentation-materials.html http://git-wip-us.apache.org/repos/asf/incubator-beam-site/blob/2302aee6/content/learn/presentation-materials/index.html ---------------------------------------------------------------------- diff --git a/content/learn/presentation-materials/index.html b/content/learn/presentation-materials/index.html index 85d5bbb..2f62961 100644 --- a/content/learn/presentation-materials/index.html +++ b/content/learn/presentation-materials/index.html @@ -158,17 +158,17 @@

Using these materials

These materials can be used by anyone. If you do use these materials, please remember to give credit to the authors cited in each presentation. These materials include speaker notes and you can view them by clicking View and selecting Show speaker notes. Some presentations have [Long] or [Short] in their name, such as the Apache Beam Model, which means there are two different presentations (a long and short version) depending on your needs and time constraints.

-

If you have any questions about the these materials, please ask on the user@beam.incubator.apache.org mailing list.

+

If you have any questions about the these materials, please ask on the user@beam.incubator.apache.org mailing list.

-

The Apache Beam community is looking for feedback on these materials to improve them over time. If you deliver a presentation based on these materials, we’d love to hear from you on the user@beam.incubator.apache.org mailing list. Please share your thoughts, questions you were asked, and (if available and appropriate) a link to the recording of your talk so the rest of the Apache Beam community can benefit from it.

+

The Apache Beam community is looking for feedback on these materials to improve them over time. If you deliver a presentation based on these materials, we’d love to hear from you on the user@beam.incubator.apache.org mailing list. Please share your thoughts, questions you were asked, and (if available and appropriate) a link to the recording of your talk so the rest of the Apache Beam community can benefit from it.

Contributing new content

The Apache Beam community is always looking for new core content and contributions!

-

If you are interested in contributing new content, we recommend you first search the Apache Beam JIRA to see if your idea currently has an open JIRA item. If it does not, send an email to the user@beam.incubator.apache.org mailing list or open a new issue in the Apache Beam JIRA using the website component. Based on community submissions or contributions, the Beam community collaboratively reviews or creates new content and then adds it to this shared space for everyone to use.

+

If you are interested in contributing new content, we recommend you first search the Apache Beam JIRA to see if your idea currently has an open JIRA item. If it does not, send an email to the user@beam.incubator.apache.org mailing list or open a new issue in the Apache Beam JIRA using the website component. Based on community submissions or contributions, the Beam community collaboratively reviews or creates new content and then adds it to this shared space for everyone to use.

Corrections and updates

-

If you encounter missing, incorrect, or out of date material, please make a comment in the presentation or send an email to the user@beam.incubator.apache.org mailing list.

+

If you encounter missing, incorrect, or out of date material, please make a comment in the presentation or send an email to the user@beam.incubator.apache.org mailing list.

http://git-wip-us.apache.org/repos/asf/incubator-beam-site/blob/2302aee6/content/learn/programming-guide/index.html ---------------------------------------------------------------------- diff --git a/content/learn/programming-guide/index.html b/content/learn/programming-guide/index.html index 8d2f5c1..18e1fa6 100644 --- a/content/learn/programming-guide/index.html +++ b/content/learn/programming-guide/index.html @@ -296,7 +296,7 @@ } -

### PCollection Characteristics

+

PCollection Characteristics

A PCollection is owned by the specific Pipeline object for which it is created; multiple pipelines cannot share a PCollection. In some respects, a PCollection functions like a collection class. However, a PCollection can differ in a few key ways:

http://git-wip-us.apache.org/repos/asf/incubator-beam-site/blob/2302aee6/content/learn/runners/capability-matrix/index.html ---------------------------------------------------------------------- diff --git a/content/learn/runners/capability-matrix/index.html b/content/learn/runners/capability-matrix/index.html index 2c7a606..debddb1 100644 --- a/content/learn/runners/capability-matrix/index.html +++ b/content/learn/runners/capability-matrix/index.html @@ -146,7 +146,7 @@

Beam Capability Matrix

-

Last updated: 2016-10-20 17:09 PDT

+

Last updated: 2016-10-25 21:24 PDT

Apache Beam (incubating) provides a portable API layer for building sophisticated data-parallel processing engines that may be executed across a diversity of exeuction engines, or runners. The core concepts of this layer are based upon the Beam Model (formerly referred to as the Dataflow Model), and implemented to varying degrees in each Beam runner. To help clarify the capabilities of individual runners, we’ve created the capability matrix below.

http://git-wip-us.apache.org/repos/asf/incubator-beam-site/blob/2302aee6/content/project/public-meetings/index.html ---------------------------------------------------------------------- diff --git a/content/project/public-meetings/index.html b/content/project/public-meetings/index.html index 78417f6..345ae3a 100644 --- a/content/project/public-meetings/index.html +++ b/content/project/public-meetings/index.html @@ -201,20 +201,20 @@

I want to give a public talk about Apache Beam

To get started, we recommend you review the Apache Beam presentation materials page to review the content the Apache Beam community has already created. These materials will possibly save you time and energy as you create content for your event.

-

Once you have scheduled your event, we welcome you to announce it on the user@beam.incubator.apache.org mailing list. Additionally, please open a JIRA item using the website component with details so we can update this page.

+

Once you have scheduled your event, we welcome you to announce it on the user@beam.incubator.apache.org mailing list. Additionally, please open a JIRA item using the website component with details so we can update this page.

-

If you have any questions as you prepare for your event, we recommend you reach out to the Apache Beam community through the dev@beam.incubator.apache.org mailing list. The Beam community can help provide feedback on your materials and promote your event.

+

If you have any questions as you prepare for your event, we recommend you reach out to the Apache Beam community through the dev@beam.incubator.apache.org mailing list. The Beam community can help provide feedback on your materials and promote your event.

Frequently asked questions about public meetings

How can I remotely attend these meetings?

-

We are investigating methods, such as video conferencing and IRC chat, to allow anyone to attend remotely. At present we do not have an estimated date. If you are interested in this option, please send an email to the user@beam.incubator.apache.org with your ideas and recommendations.

+

We are investigating methods, such as video conferencing and IRC chat, to allow anyone to attend remotely. At present we do not have an estimated date. If you are interested in this option, please send an email to the user@beam.incubator.apache.org with your ideas and recommendations.

What is a public meeting?

Public meetings include scheduled Apache Beam Dev/PPMC meetings, Meetup events, conference talks, and other events where the public meets to discuss Beam.

How do I learn about new meetings?

-

The Apache Beam community announces upcoming public meetings on the dev@beam.incubator.apache.org mailing list. If you want to learn about new events, we recommend you subscribe to that list. If you are holding a public event, please send an email to the dev@ list.

+

The Apache Beam community announces upcoming public meetings on the dev@beam.incubator.apache.org mailing list. If you want to learn about new events, we recommend you subscribe to that list. If you are holding a public event, please send an email to the dev@ list.

http://git-wip-us.apache.org/repos/asf/incubator-beam-site/blob/2302aee6/content/styles/site.css ---------------------------------------------------------------------- diff --git a/content/styles/site.css b/content/styles/site.css index afba989..0aa93ec 100644 --- a/content/styles/site.css +++ b/content/styles/site.css @@ -5809,6 +5809,146 @@ button.close { @media print { .hidden-print { display: none !important; } } +/** + * Syntax highlighting styles + */ +.highlight { + background: #fff; } + .highlighter-rouge .highlight { + background: #eef; } + .highlight .c { + color: #998; + font-style: italic; } + .highlight .err { + color: #a61717; } + .highlight .k { + font-weight: bold; } + .highlight .o { + font-weight: bold; } + .highlight .cm { + color: #998; + font-style: italic; } + .highlight .cp { + color: #999; + font-weight: bold; } + .highlight .c1 { + color: #998; + font-style: italic; } + .highlight .cs { + color: #999; + font-weight: bold; + font-style: italic; } + .highlight .gd { + color: #000; + background-color: #fdd; } + .highlight .gd .x { + color: #000; + background-color: #faa; } + .highlight .ge { + font-style: italic; } + .highlight .gr { + color: #a00; } + .highlight .gh { + color: #999; } + .highlight .gi { + color: #000; + background-color: #dfd; } + .highlight .gi .x { + color: #000; + background-color: #afa; } + .highlight .go { + color: #888; } + .highlight .gp { + color: #555; } + .highlight .gs { + font-weight: bold; } + .highlight .gu { + color: #aaa; } + .highlight .gt { + color: #a00; } + .highlight .kc { + font-weight: bold; } + .highlight .kd { + font-weight: bold; } + .highlight .kp { + font-weight: bold; } + .highlight .kr { + font-weight: bold; } + .highlight .kt { + color: #458; + font-weight: bold; } + .highlight .m { + color: #099; } + .highlight .s { + color: #d14; } + .highlight .na { + color: #008080; } + .highlight .nb { + color: #0086B3; } + .highlight .nc { + color: #458; + font-weight: bold; } + .highlight .no { + color: #008080; } + .highlight .ni { + color: #800080; } + .highlight .ne { + color: #900; + font-weight: bold; } + .highlight .nf { + color: #900; + font-weight: bold; } + .highlight .nn { + color: #555; } + .highlight .nt { + color: #000080; } + .highlight .nv { + color: #008080; } + .highlight .ow { + font-weight: bold; } + .highlight .w { + color: #bbb; } + .highlight .mf { + color: #099; } + .highlight .mh { + color: #099; } + .highlight .mi { + color: #099; } + .highlight .mo { + color: #099; } + .highlight .sb { + color: #d14; } + .highlight .sc { + color: #d14; } + .highlight .sd { + color: #d14; } + .highlight .s2 { + color: #d14; } + .highlight .se { + color: #d14; } + .highlight .sh { + color: #d14; } + .highlight .si { + color: #d14; } + .highlight .sx { + color: #d14; } + .highlight .sr { + color: #009926; } + .highlight .s1 { + color: #d14; } + .highlight .ss { + color: #990073; } + .highlight .bp { + color: #999; } + .highlight .vc { + color: #008080; } + .highlight .vg { + color: #008080; } + .highlight .vi { + color: #008080; } + .highlight .il { + color: #099; } + /* What/Where/When/How colors. */ .wwwh-what-dark { color: #ca1;