Return-Path: X-Original-To: archive-asf-public-internal@cust-asf2.ponee.io Delivered-To: archive-asf-public-internal@cust-asf2.ponee.io Received: from cust-asf.ponee.io (cust-asf.ponee.io [163.172.22.183]) by cust-asf2.ponee.io (Postfix) with ESMTP id 76E3D200C4F for ; Sat, 18 Mar 2017 02:35:31 +0100 (CET) Received: by cust-asf.ponee.io (Postfix) id 756ED160B80; Sat, 18 Mar 2017 01:35:31 +0000 (UTC) Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by cust-asf.ponee.io (Postfix) with SMTP id A0BDD160B92 for ; Sat, 18 Mar 2017 02:35:29 +0100 (CET) Received: (qmail 22575 invoked by uid 500); 18 Mar 2017 01:35:28 -0000 Mailing-List: contact commits-help@beam.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@beam.apache.org Delivered-To: mailing list commits@beam.apache.org Received: (qmail 22467 invoked by uid 99); 18 Mar 2017 01:35:28 -0000 Received: from git1-us-west.apache.org (HELO git1-us-west.apache.org) (140.211.11.23) by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 18 Mar 2017 01:35:28 +0000 Received: by git1-us-west.apache.org (ASF Mail Server at git1-us-west.apache.org, from userid 33) id 7EAA4DFE7B; Sat, 18 Mar 2017 01:35:28 +0000 (UTC) Content-Type: text/plain; charset="us-ascii" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit From: davor@apache.org To: commits@beam.apache.org Date: Sat, 18 Mar 2017 01:35:29 -0000 Message-Id: <3c427f79bd0f4fb1979abb85afc74b86@git.apache.org> In-Reply-To: <8902bca714b64e13bb01e419cc5ff32a@git.apache.org> References: <8902bca714b64e13bb01e419cc5ff32a@git.apache.org> X-Mailer: ASF-Git Admin Mailer Subject: [2/4] beam-site git commit: Regenerate website archived-at: Sat, 18 Mar 2017 01:35:31 -0000 http://git-wip-us.apache.org/repos/asf/beam-site/blob/35c2d97b/content/documentation/io/contributing/index.html ---------------------------------------------------------------------- diff --git a/content/documentation/io/contributing/index.html b/content/documentation/io/contributing/index.html new file mode 100644 index 0000000..43ace0c --- /dev/null +++ b/content/documentation/io/contributing/index.html @@ -0,0 +1,193 @@ + + + + + + + + + Contributing I/O Transforms + + + + + + + + + + + + + + + + + + + + + + +
+ +
+

Pipeline I/O Table of Contents

+ +

Contributing I/O Transforms

+ +
    +
  • If you are planning to contribute your I/O transform to the Apache Beam community, you’ll be going through the normal Beam contribution life cycle - see the Apache Beam Contribution Guide for more details.
    • +
  • Talk to the community!
    • +
  • Make sure you’ve implemented the appropriate tests as discussed in the Testing I/O Transforms section.
    • +
+ +
+

Note: This guide is still in progress. There is an open issue to finish the guide: BEAM-1025.

+
+ +
+ + +
+
+
+ +
+
+ +
+ + + + + http://git-wip-us.apache.org/repos/asf/beam-site/blob/35c2d97b/content/documentation/io/io-toc/index.html ---------------------------------------------------------------------- diff --git a/content/documentation/io/io-toc/index.html b/content/documentation/io/io-toc/index.html new file mode 100644 index 0000000..3e35922 --- /dev/null +++ b/content/documentation/io/io-toc/index.html @@ -0,0 +1,203 @@ + + + + + + + + + Pipeline I/O + + + + + + + + + + + + + + + + + + + + + + +
+ +
+

Pipeline I/O

+ +

Using Pipeline I/O

+ + +

Authoring Read & Write I/O Transforms

+ +
+

Note: This guide is still in progress. There is an open issue to finish the guide: BEAM-1025.

+
+ + + +
+ + +
+
+
+ +
+
+ +
+ + + + + http://git-wip-us.apache.org/repos/asf/beam-site/blob/35c2d97b/content/documentation/io/testing/index.html ---------------------------------------------------------------------- diff --git a/content/documentation/io/testing/index.html b/content/documentation/io/testing/index.html new file mode 100644 index 0000000..03ba859 --- /dev/null +++ b/content/documentation/io/testing/index.html @@ -0,0 +1,194 @@ + + + + + + + + + Testing I/O Transforms + + + + + + + + + + + + + + + + + + + + + + +
+ +
+

Pipeline I/O Table of Contents

+ +

Testing I/O Transforms

+ +
+

Note: This guide is still in progress. There is an open issue to finish the guide: BEAM-1025.

+
+ +

Next steps

+ +

If you have a well tested I/O transform, why not contribute it to Apache Beam? Read all about it:

+ +

Contributing I/O Transforms

+ + +
+ + +
+
+
+ +
+
+ +
+ + + + + http://git-wip-us.apache.org/repos/asf/beam-site/blob/35c2d97b/content/documentation/pipelines/create-your-pipeline/index.html ---------------------------------------------------------------------- diff --git a/content/documentation/pipelines/create-your-pipeline/index.html b/content/documentation/pipelines/create-your-pipeline/index.html index a8bca14..8911488 100644 --- a/content/documentation/pipelines/create-your-pipeline/index.html +++ b/content/documentation/pipelines/create-your-pipeline/index.html @@ -77,6 +77,7 @@
  • Design Your Pipeline
  • Create Your Pipeline
  • Test Your Pipeline
    • +
  • Pipeline I/O
  • SDKs
  • Java SDK
    • http://git-wip-us.apache.org/repos/asf/beam-site/blob/35c2d97b/content/documentation/pipelines/design-your-pipeline/index.html ---------------------------------------------------------------------- diff --git a/content/documentation/pipelines/design-your-pipeline/index.html b/content/documentation/pipelines/design-your-pipeline/index.html index 84186c8..c575446 100644 --- a/content/documentation/pipelines/design-your-pipeline/index.html +++ b/content/documentation/pipelines/design-your-pipeline/index.html @@ -77,6 +77,7 @@
  • Design Your Pipeline
  • Create Your Pipeline
  • Test Your Pipeline
    • +
  • Pipeline I/O
  • SDKs
  • Java SDK
    • http://git-wip-us.apache.org/repos/asf/beam-site/blob/35c2d97b/content/documentation/pipelines/test-your-pipeline/index.html ---------------------------------------------------------------------- diff --git a/content/documentation/pipelines/test-your-pipeline/index.html b/content/documentation/pipelines/test-your-pipeline/index.html index 022cae5..112700b 100644 --- a/content/documentation/pipelines/test-your-pipeline/index.html +++ b/content/documentation/pipelines/test-your-pipeline/index.html @@ -77,6 +77,7 @@
  • Design Your Pipeline
  • Create Your Pipeline
  • Test Your Pipeline
    • +
  • Pipeline I/O
  • SDKs
  • Java SDK
    • http://git-wip-us.apache.org/repos/asf/beam-site/blob/35c2d97b/content/documentation/programming-guide/index.html ---------------------------------------------------------------------- diff --git a/content/documentation/programming-guide/index.html b/content/documentation/programming-guide/index.html index 707c525..19853df 100644 --- a/content/documentation/programming-guide/index.html +++ b/content/documentation/programming-guide/index.html @@ -77,6 +77,7 @@
  • Design Your Pipeline
  • Create Your Pipeline
  • Test Your Pipeline
    • +
  • Pipeline I/O
  • SDKs
  • Java SDK
    • @@ -1220,11 +1221,7 @@ tree, [2]

    Pipeline I/O

    -

    When you create a pipeline, you often need to read data from some external source, such as a file in external data sink or a database. Likewise, you may want your pipeline to output its result data to a similar external data sink. Beam provides read and write transforms for a number of common data storage types. If you want your pipeline to read from or write to a data storage format that isn’t supported by the built-in transforms, you can implement your own read and write transforms.

    - -
    -

    A guide that covers how to implement your own Beam IO transforms is in progress (BEAM-1025).

    -
    +

    When you create a pipeline, you often need to read data from some external source, such as a file in external data sink or a database. Likewise, you may want your pipeline to output its result data to a similar external data sink. Beam provides read and write transforms for a number of common data storage types. If you want your pipeline to read from or write to a data storage format that isn’t supported by the built-in transforms, you can implement your own read and write transforms.

    Reading input data

    @@ -1290,55 +1287,8 @@ tree, [2] -

    Beam-provided I/O APIs

    - -

    See the language specific source code directories for the Beam supported I/O APIs. Specific documentation for each of these I/O sources will be added in the future. (BEAM-1054)

    - - - - - - - - - - - - - - - - - - - - - -
    LanguageFile-basedMessagingDatabase
    Java -

    AvroIO

    -

    HDFS

    -

    TextIO

    -

    XML

    -     		-

    JMS

    -

    Kafka

    -

    Kinesis

    -

    Google Cloud PubSub

    -     		-

    Apache HBase

    -

    MongoDB

    -

    JDBC

    -

    Google BigQuery

    -

    Google Cloud Bigtable

    -

    Google Cloud Datastore

    -
    Python -

    avroio

    -

    textio

    -     		- -

    Google BigQuery

    -

    Google Cloud Datastore

    -
    +

    Beam-provided I/O Transforms

    +

    See the Beam-provided I/O Transforms page for a list of the currently available I/O transforms.

    Running the pipeline

    http://git-wip-us.apache.org/repos/asf/beam-site/blob/35c2d97b/content/documentation/resources/index.html ---------------------------------------------------------------------- diff --git a/content/documentation/resources/index.html b/content/documentation/resources/index.html index ae6c691..dc74031 100644 --- a/content/documentation/resources/index.html +++ b/content/documentation/resources/index.html @@ -77,6 +77,7 @@
  • Design Your Pipeline
  • Create Your Pipeline
  • Test Your Pipeline
    • +
  • Pipeline I/O
  • SDKs
  • Java SDK
    • http://git-wip-us.apache.org/repos/asf/beam-site/blob/35c2d97b/content/documentation/runners/apex/index.html ---------------------------------------------------------------------- diff --git a/content/documentation/runners/apex/index.html b/content/documentation/runners/apex/index.html index f85c270..875f3e0 100644 --- a/content/documentation/runners/apex/index.html +++ b/content/documentation/runners/apex/index.html @@ -77,6 +77,7 @@
  • Design Your Pipeline
  • Create Your Pipeline
  • Test Your Pipeline
    • +
  • Pipeline I/O
  • SDKs
  • Java SDK
    • http://git-wip-us.apache.org/repos/asf/beam-site/blob/35c2d97b/content/documentation/runners/capability-matrix/index.html ---------------------------------------------------------------------- diff --git a/content/documentation/runners/capability-matrix/index.html b/content/documentation/runners/capability-matrix/index.html index efce223..04174c7 100644 --- a/content/documentation/runners/capability-matrix/index.html +++ b/content/documentation/runners/capability-matrix/index.html @@ -77,6 +77,7 @@
  • Design Your Pipeline
  • Create Your Pipeline
  • Test Your Pipeline
    • +
  • Pipeline I/O
  • SDKs
  • Java SDK
    • http://git-wip-us.apache.org/repos/asf/beam-site/blob/35c2d97b/content/documentation/runners/dataflow/index.html ---------------------------------------------------------------------- diff --git a/content/documentation/runners/dataflow/index.html b/content/documentation/runners/dataflow/index.html index 259e99f..2f3d9b0 100644 --- a/content/documentation/runners/dataflow/index.html +++ b/content/documentation/runners/dataflow/index.html @@ -77,6 +77,7 @@
  • Design Your Pipeline
  • Create Your Pipeline
  • Test Your Pipeline
    • +
  • Pipeline I/O
  • SDKs
  • Java SDK
    • http://git-wip-us.apache.org/repos/asf/beam-site/blob/35c2d97b/content/documentation/runners/direct/index.html ---------------------------------------------------------------------- diff --git a/content/documentation/runners/direct/index.html b/content/documentation/runners/direct/index.html index 11bbf6e..dd1151a 100644 --- a/content/documentation/runners/direct/index.html +++ b/content/documentation/runners/direct/index.html @@ -77,6 +77,7 @@
  • Design Your Pipeline
  • Create Your Pipeline
  • Test Your Pipeline
    • +
  • Pipeline I/O
  • SDKs
  • Java SDK
    • http://git-wip-us.apache.org/repos/asf/beam-site/blob/35c2d97b/content/documentation/runners/flink/index.html ---------------------------------------------------------------------- diff --git a/content/documentation/runners/flink/index.html b/content/documentation/runners/flink/index.html index f451eba..b0b8fdc 100644 --- a/content/documentation/runners/flink/index.html +++ b/content/documentation/runners/flink/index.html @@ -77,6 +77,7 @@
  • Design Your Pipeline
  • Create Your Pipeline
  • Test Your Pipeline
    • +
  • Pipeline I/O
  • SDKs
  • Java SDK
    • http://git-wip-us.apache.org/repos/asf/beam-site/blob/35c2d97b/content/documentation/runners/spark/index.html ---------------------------------------------------------------------- diff --git a/content/documentation/runners/spark/index.html b/content/documentation/runners/spark/index.html index 29fce06..16c3e5b 100644 --- a/content/documentation/runners/spark/index.html +++ b/content/documentation/runners/spark/index.html @@ -77,6 +77,7 @@
  • Design Your Pipeline
  • Create Your Pipeline
  • Test Your Pipeline
    • +
  • Pipeline I/O
  • SDKs
  • Java SDK
    • http://git-wip-us.apache.org/repos/asf/beam-site/blob/35c2d97b/content/documentation/sdks/java/index.html ---------------------------------------------------------------------- diff --git a/content/documentation/sdks/java/index.html b/content/documentation/sdks/java/index.html index 4901fb3..18f406d 100644 --- a/content/documentation/sdks/java/index.html +++ b/content/documentation/sdks/java/index.html @@ -77,6 +77,7 @@
  • Design Your Pipeline
  • Create Your Pipeline
  • Test Your Pipeline
    • +
  • Pipeline I/O
  • SDKs
  • Java SDK
    • @@ -164,26 +165,8 @@

    The Java SDK supports all features currently supported by the Beam model.

    -

    Supported IO Connectors

    - -
      -
    • Amazon Kinesis
      • -
    • Apache Hadoop’s FileInputFormat in Hadoop Distributed File System (HDFS)
      • -
    • Apache HBase
      • -
    • Apache Kafka
      • -
    • Avro Files
      • -
    • Google BigQuery
      • -
    • Google Cloud Bigtable
      • -
    • Google Cloud Datastore
      • -
    • Google Cloud Pub/Sub
      • -
    • Google Cloud Storage
      • -
    • Java Database Connectivity (JDBC)
      • -
    • Java Message Service (JMS)
      • -
    • MongoDB
      • -
    • Text Files
      • -
    • XML Files
      • -
    - +

    Pipeline I/O

    +

    See the Beam-provided I/O Transforms page for a list of the currently available I/O transforms.

    http://git-wip-us.apache.org/repos/asf/beam-site/blob/35c2d97b/content/documentation/sdks/python-custom-io/index.html ---------------------------------------------------------------------- diff --git a/content/documentation/sdks/python-custom-io/index.html b/content/documentation/sdks/python-custom-io/index.html index d6195b6..d078e33 100644 --- a/content/documentation/sdks/python-custom-io/index.html +++ b/content/documentation/sdks/python-custom-io/index.html @@ -77,6 +77,7 @@
  • Design Your Pipeline
  • Create Your Pipeline
  • Test Your Pipeline
    • +
  • Pipeline I/O
  • SDKs
  • Java SDK
    • http://git-wip-us.apache.org/repos/asf/beam-site/blob/35c2d97b/content/documentation/sdks/python-pipeline-dependencies/index.html ---------------------------------------------------------------------- diff --git a/content/documentation/sdks/python-pipeline-dependencies/index.html b/content/documentation/sdks/python-pipeline-dependencies/index.html index c0d9bdd..dba6ff7 100644 --- a/content/documentation/sdks/python-pipeline-dependencies/index.html +++ b/content/documentation/sdks/python-pipeline-dependencies/index.html @@ -77,6 +77,7 @@
  • Design Your Pipeline
  • Create Your Pipeline
  • Test Your Pipeline
    • +
  • Pipeline I/O
  • SDKs
  • Java SDK
    • http://git-wip-us.apache.org/repos/asf/beam-site/blob/35c2d97b/content/documentation/sdks/python-type-safety/index.html ---------------------------------------------------------------------- diff --git a/content/documentation/sdks/python-type-safety/index.html b/content/documentation/sdks/python-type-safety/index.html index 092e02f..24900b8 100644 --- a/content/documentation/sdks/python-type-safety/index.html +++ b/content/documentation/sdks/python-type-safety/index.html @@ -77,6 +77,7 @@
  • Design Your Pipeline
  • Create Your Pipeline
  • Test Your Pipeline
    • +
  • Pipeline I/O
  • SDKs
  • Java SDK
    • http://git-wip-us.apache.org/repos/asf/beam-site/blob/35c2d97b/content/documentation/sdks/python/index.html ---------------------------------------------------------------------- diff --git a/content/documentation/sdks/python/index.html b/content/documentation/sdks/python/index.html index d7f7240..e5c9db7 100644 --- a/content/documentation/sdks/python/index.html +++ b/content/documentation/sdks/python/index.html @@ -77,6 +77,7 @@
  • Design Your Pipeline
  • Create Your Pipeline
  • Test Your Pipeline
    • +
  • Pipeline I/O
  • SDKs
  • Java SDK
    • http://git-wip-us.apache.org/repos/asf/beam-site/blob/35c2d97b/content/get-started/beam-overview/index.html ---------------------------------------------------------------------- diff --git a/content/get-started/beam-overview/index.html b/content/get-started/beam-overview/index.html index 7a2b1f1..caac276 100644 --- a/content/get-started/beam-overview/index.html +++ b/content/get-started/beam-overview/index.html @@ -77,6 +77,7 @@
  • Design Your Pipeline
  • Create Your Pipeline
  • Test Your Pipeline
    • +
  • Pipeline I/O
  • SDKs
  • Java SDK
    • http://git-wip-us.apache.org/repos/asf/beam-site/blob/35c2d97b/content/get-started/downloads/index.html ---------------------------------------------------------------------- diff --git a/content/get-started/downloads/index.html b/content/get-started/downloads/index.html index 3438ee7..9aedfeb 100644 --- a/content/get-started/downloads/index.html +++ b/content/get-started/downloads/index.html @@ -77,6 +77,7 @@
  • Design Your Pipeline
  • Create Your Pipeline
  • Test Your Pipeline
    • +
  • Pipeline I/O
  • SDKs
  • Java SDK
    • http://git-wip-us.apache.org/repos/asf/beam-site/blob/35c2d97b/content/get-started/index.html ---------------------------------------------------------------------- diff --git a/content/get-started/index.html b/content/get-started/index.html index b7cc440..b33241c 100644 --- a/content/get-started/index.html +++ b/content/get-started/index.html @@ -77,6 +77,7 @@
  • Design Your Pipeline
  • Create Your Pipeline
  • Test Your Pipeline
    • +
  • Pipeline I/O
  • SDKs
  • Java SDK
    • http://git-wip-us.apache.org/repos/asf/beam-site/blob/35c2d97b/content/get-started/mobile-gaming-example/index.html ---------------------------------------------------------------------- diff --git a/content/get-started/mobile-gaming-example/index.html b/content/get-started/mobile-gaming-example/index.html index 0489219..5b7e487 100644 --- a/content/get-started/mobile-gaming-example/index.html +++ b/content/get-started/mobile-gaming-example/index.html @@ -77,6 +77,7 @@
  • Design Your Pipeline
  • Create Your Pipeline
  • Test Your Pipeline
    • +
  • Pipeline I/O
  • SDKs
  • Java SDK
    • http://git-wip-us.apache.org/repos/asf/beam-site/blob/35c2d97b/content/get-started/quickstart-java/index.html ---------------------------------------------------------------------- diff --git a/content/get-started/quickstart-java/index.html b/content/get-started/quickstart-java/index.html index d4b6053..51f5e4c 100644 --- a/content/get-started/quickstart-java/index.html +++ b/content/get-started/quickstart-java/index.html @@ -77,6 +77,7 @@
  • Design Your Pipeline
  • Create Your Pipeline
  • Test Your Pipeline
    • +
  • Pipeline I/O
  • SDKs
  • Java SDK
    • http://git-wip-us.apache.org/repos/asf/beam-site/blob/35c2d97b/content/get-started/quickstart-py/index.html ---------------------------------------------------------------------- diff --git a/content/get-started/quickstart-py/index.html b/content/get-started/quickstart-py/index.html index 153eac6..b717987 100644 --- a/content/get-started/quickstart-py/index.html +++ b/content/get-started/quickstart-py/index.html @@ -77,6 +77,7 @@
  • Design Your Pipeline
  • Create Your Pipeline
  • Test Your Pipeline
    • +
  • Pipeline I/O
  • SDKs
  • Java SDK
    • http://git-wip-us.apache.org/repos/asf/beam-site/blob/35c2d97b/content/get-started/support/index.html ---------------------------------------------------------------------- diff --git a/content/get-started/support/index.html b/content/get-started/support/index.html index 9a00712..bf9702c 100644 --- a/content/get-started/support/index.html +++ b/content/get-started/support/index.html @@ -77,6 +77,7 @@
  • Design Your Pipeline
  • Create Your Pipeline
  • Test Your Pipeline
    • +
  • Pipeline I/O
  • SDKs
  • Java SDK
    • http://git-wip-us.apache.org/repos/asf/beam-site/blob/35c2d97b/content/get-started/wordcount-example/index.html ---------------------------------------------------------------------- diff --git a/content/get-started/wordcount-example/index.html b/content/get-started/wordcount-example/index.html index a5be22d..030c3e2 100644 --- a/content/get-started/wordcount-example/index.html +++ b/content/get-started/wordcount-example/index.html @@ -77,6 +77,7 @@
  • Design Your Pipeline
  • Create Your Pipeline
  • Test Your Pipeline
    • +
  • Pipeline I/O
  • SDKs
  • Java SDK
    • http://git-wip-us.apache.org/repos/asf/beam-site/blob/35c2d97b/content/index.html ---------------------------------------------------------------------- diff --git a/content/index.html b/content/index.html index dcb2f23..7f896cf 100644 --- a/content/index.html +++ b/content/index.html @@ -77,6 +77,7 @@
  • Design Your Pipeline
  • Create Your Pipeline
  • Test Your Pipeline
    • +
  • Pipeline I/O
  • SDKs
  • Java SDK
    • http://git-wip-us.apache.org/repos/asf/beam-site/blob/35c2d97b/content/privacy_policy/index.html ---------------------------------------------------------------------- diff --git a/content/privacy_policy/index.html b/content/privacy_policy/index.html index 951f387..eaef599 100644 --- a/content/privacy_policy/index.html +++ b/content/privacy_policy/index.html @@ -77,6 +77,7 @@
  • Design Your Pipeline
  • Create Your Pipeline
  • Test Your Pipeline
    • +
  • Pipeline I/O
  • SDKs
  • Java SDK