Return-Path: X-Original-To: archive-asf-public-internal@cust-asf2.ponee.io Delivered-To: archive-asf-public-internal@cust-asf2.ponee.io Received: from cust-asf.ponee.io (cust-asf.ponee.io [163.172.22.183]) by cust-asf2.ponee.io (Postfix) with ESMTP id D01C3200B64 for ; Tue, 2 Aug 2016 19:46:57 +0200 (CEST) Received: by cust-asf.ponee.io (Postfix) id CEB10160A65; Tue, 2 Aug 2016 17:46:57 +0000 (UTC) Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by cust-asf.ponee.io (Postfix) with SMTP id 5131C160AA8 for ; Tue, 2 Aug 2016 19:46:55 +0200 (CEST) Received: (qmail 12978 invoked by uid 500); 2 Aug 2016 17:46:54 -0000 Mailing-List: contact commits-help@beam.incubator.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@beam.incubator.apache.org Delivered-To: mailing list commits@beam.incubator.apache.org Received: (qmail 12969 invoked by uid 99); 2 Aug 2016 17:46:54 -0000 Received: from pnap-us-west-generic-nat.apache.org (HELO spamd4-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 02 Aug 2016 17:46:54 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd4-us-west.apache.org (ASF Mail Server at spamd4-us-west.apache.org) with ESMTP id D1DFAC0957 for ; Tue, 2 Aug 2016 17:46:53 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd4-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: -4.646 X-Spam-Level: X-Spam-Status: No, score=-4.646 tagged_above=-999 required=6.31 tests=[KAM_ASCII_DIVIDERS=0.8, KAM_LAZY_DOMAIN_SECURITY=1, RCVD_IN_DNSWL_HI=-5, RCVD_IN_MSPIKE_H3=-0.01, RCVD_IN_MSPIKE_WL=-0.01, RP_MATCHES_RCVD=-1.426] autolearn=disabled Received: from mx1-lw-eu.apache.org ([10.40.0.8]) by localhost (spamd4-us-west.apache.org [10.40.0.11]) (amavisd-new, port 10024) with ESMTP id mXuJ4_w1Cw5h for ; Tue, 2 Aug 2016 17:46:40 +0000 (UTC) Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by mx1-lw-eu.apache.org (ASF Mail Server at mx1-lw-eu.apache.org) with SMTP id D53ED5FB8F for ; Tue, 2 Aug 2016 17:46:33 +0000 (UTC) Received: (qmail 9706 invoked by uid 99); 2 Aug 2016 17:46:33 -0000 Received: from git1-us-west.apache.org (HELO git1-us-west.apache.org) (140.211.11.23) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 02 Aug 2016 17:46:33 +0000 Received: by git1-us-west.apache.org (ASF Mail Server at git1-us-west.apache.org, from userid 33) id 96BAFEE696; Tue, 2 Aug 2016 17:46:32 +0000 (UTC) Content-Type: text/plain; charset="us-ascii" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit From: dhalperi@apache.org To: commits@beam.incubator.apache.org Date: Tue, 02 Aug 2016 17:46:39 -0000 Message-Id: <5536fbff5d944e8aae52fba25ede8730@git.apache.org> In-Reply-To: <2d6838177b24473b83cd23bb0ff68f23@git.apache.org> References: <2d6838177b24473b83cd23bb0ff68f23@git.apache.org> X-Mailer: ASF-Git Admin Mailer Subject: [08/15] incubator-beam-site git commit: Regenerated site after merge. archived-at: Tue, 02 Aug 2016 17:46:58 -0000 http://git-wip-us.apache.org/repos/asf/incubator-beam-site/blob/8797d044/content/learn/runners/capability-matrix/index.html ---------------------------------------------------------------------- diff --git a/content/learn/runners/capability-matrix/index.html b/content/learn/runners/capability-matrix/index.html new file mode 100644 index 0000000..28ed044 --- /dev/null +++ b/content/learn/runners/capability-matrix/index.html @@ -0,0 +1,1716 @@ + + + + + + + + + Apache Beam Capability Matrix + + + + + + + + + + + + + + + + + + + + + +
+ +
+

Beam Capability Matrix

+

Last updated: 2016-08-02 10:42 PDT

+ +

Apache Beam (incubating) provides a portable API layer for building sophisticated data-parallel processing engines that may be executed across a diversity of exeuction engines, or runners. The core concepts of this layer are based upon the Beam Model (formerly referred to as the Dataflow Model), and implemented to varying degrees in each Beam runner. To help clarify the capabilities of individual runners, we’ve created the capability matrix below.

+ +

Individual capabilities have been grouped by their corresponding What / Where / When / How question:

+ +
    +
  • What results are being calculated?
    • +
  • Where in event time?
    • +
  • When in processing time?
    • +
  • How do refinements of results relate?
    • +
+ +

For more details on the What / Where / When / How breakdown of concepts, we recommend reading through the Streaming 102 post on O’Reilly Radar.

+ +

Note that in the future, we intend to add additional tables beyond the current set, for things like runtime characterstics (e.g. at-least-once vs exactly-once), performance, etc.

+ + + +
+ + + + + + + + +
+ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
(expand details)
What is being computed?
Beam ModelGoogle Cloud DataflowApache FlinkApache Spark
ParDo
✓
+
✓
+
✓
+
✓
+
GroupByKey
✓
+
✓
+
✓
+
~
+
Flatten
✓
+
✓
+
✓
+
✓
+
Combine
✓
+
✓
+
✓
+
✓
+
Composite Transforms
✓
+
~
+
~
+
~
+
Side Inputs
✓
+
✓
+
~ (BEAM-102)
+
~
+
Source API
✓
+
✓
+
✓
+
✓
+
Aggregators
~
+
~
+
~
+
~
+
Keyed State
✕ (BEAM-25)
+
✕
+
✕
+
✕
+
(expand details)
Where in event time?
Beam ModelGoogle Cloud DataflowApache FlinkApache Spark
Global windows
✓
+
✓
+
✓
+
✓
+
Fixed windows
✓
+
✓
+
✓
+
~
+
Sliding windows
✓
+
✓
+
✓
+
~
+
Session windows
✓
+
✓
+
✓
+
✕
+
Custom windows
✓
+
✓
+
✓
+
✕
+
Custom merging windows
✓
+
✓
+
✓
+
✕
+
Timestamp control
✓
+
✓
+
✓
+
✕
+
(expand details)
When in processing time?
Beam ModelGoogle Cloud DataflowApache FlinkApache Spark
Configurable triggering
✓
+
✓
+
✓
+
✕
+
Event-time triggers
✓
+
✓
+
✓
+
✕
+
Processing-time triggers
✓
+
✓
+
✓
+
✓
+
Count triggers
✓
+
✓
+
✓
+
✕
+
[Meta]data driven triggers
✕ (BEAM-101)
+
✕
+
✕
+
✕
+
Composite triggers
✓
+
✓
+
✓
+
✕
+
Allowed lateness
✓
+
✓
+
✓
+
✕
+
Timers
✕ (BEAM-27)
+
✕
+
✕
+
✕
+
(expand details)
How do refinements relate?
Beam ModelGoogle Cloud DataflowApache FlinkApache Spark
Discarding
✓
+
✓
+
✓
+
✓
+
Accumulating
✓
+
✓
+
✓
+
✕
+
Accumulating & Retracting
✕ (BEAM-91)
+
✕
+
✕
+
✕
+
+
+ + + + + + + + + + + +
+ +
+ + +
+
+
+ +
+
+ +
+ + + + + http://git-wip-us.apache.org/repos/asf/incubator-beam-site/blob/8797d044/content/learn/runners/dataflow/index.html ---------------------------------------------------------------------- diff --git a/content/learn/runners/dataflow/index.html b/content/learn/runners/dataflow/index.html new file mode 100644 index 0000000..a45e677 --- /dev/null +++ b/content/learn/runners/dataflow/index.html @@ -0,0 +1,153 @@ + + + + + + + + + Cloud Dataflow Runner + + + + + + + + + + + + + + + + + + + + + +
+ +
+

Using the Cloud Dataflow Runner

+ +

This page is under construction (BEAM-508).

+ +
+ + +
+
+
+ +
+
+ +
+ + + + + http://git-wip-us.apache.org/repos/asf/incubator-beam-site/blob/8797d044/content/learn/runners/direct/index.html ---------------------------------------------------------------------- diff --git a/content/learn/runners/direct/index.html b/content/learn/runners/direct/index.html new file mode 100644 index 0000000..8a8ab40 --- /dev/null +++ b/content/learn/runners/direct/index.html @@ -0,0 +1,153 @@ + + + + + + + + + Apache Direct Runner + + + + + + + + + + + + + + + + + + + + + +
+ +
+

Using the Direct Runner

+ +

This page is under construction (BEAM-505).

+ +
+ + +
+
+
+ +
+
+ +
+ + + + + http://git-wip-us.apache.org/repos/asf/incubator-beam-site/blob/8797d044/content/learn/runners/flink/index.html ---------------------------------------------------------------------- diff --git a/content/learn/runners/flink/index.html b/content/learn/runners/flink/index.html new file mode 100644 index 0000000..ad28860 --- /dev/null +++ b/content/learn/runners/flink/index.html @@ -0,0 +1,153 @@ + + + + + + + + + Apache Flink Runner + + + + + + + + + + + + + + + + + + + + + +
+ +
+

Using the Apache Flink Runner

+ +

This page is under construction (BEAM-506).

+ +
+ + +
+
+
+ +
+
+ +
+ + + + +