Return-Path: X-Original-To: archive-asf-public-internal@cust-asf2.ponee.io Delivered-To: archive-asf-public-internal@cust-asf2.ponee.io Received: from cust-asf.ponee.io (cust-asf.ponee.io [163.172.22.183]) by cust-asf2.ponee.io (Postfix) with ESMTP id 7D86F200B64 for ; Tue, 2 Aug 2016 22:04:06 +0200 (CEST) Received: by cust-asf.ponee.io (Postfix) id 78D32160AA8; Tue, 2 Aug 2016 20:04:06 +0000 (UTC) Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by cust-asf.ponee.io (Postfix) with SMTP id 286D2160AAF for ; Tue, 2 Aug 2016 22:04:04 +0200 (CEST) Received: (qmail 42147 invoked by uid 500); 2 Aug 2016 20:04:03 -0000 Mailing-List: contact commits-help@beam.incubator.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@beam.incubator.apache.org Delivered-To: mailing list commits@beam.incubator.apache.org Received: (qmail 42128 invoked by uid 99); 2 Aug 2016 20:04:03 -0000 Received: from pnap-us-west-generic-nat.apache.org (HELO spamd3-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 02 Aug 2016 20:04:03 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd3-us-west.apache.org (ASF Mail Server at spamd3-us-west.apache.org) with ESMTP id B128018129E for ; Tue, 2 Aug 2016 20:04:02 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd3-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: -3.446 X-Spam-Level: X-Spam-Status: No, score=-3.446 tagged_above=-999 required=6.31 tests=[KAM_ASCII_DIVIDERS=0.8, KAM_LAZY_DOMAIN_SECURITY=1, KAM_MANYCOMMENTS=1.2, RCVD_IN_DNSWL_HI=-5, RCVD_IN_MSPIKE_H3=-0.01, RCVD_IN_MSPIKE_WL=-0.01, RP_MATCHES_RCVD=-1.426] autolearn=disabled Received: from mx1-lw-us.apache.org ([10.40.0.8]) by localhost (spamd3-us-west.apache.org [10.40.0.10]) (amavisd-new, port 10024) with ESMTP id EbXnFHxQdtkE for ; Tue, 2 Aug 2016 20:03:49 +0000 (UTC) Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by mx1-lw-us.apache.org (ASF Mail Server at mx1-lw-us.apache.org) with SMTP id AFD6A5FD68 for ; Tue, 2 Aug 2016 20:03:48 +0000 (UTC) Received: (qmail 41417 invoked by uid 99); 2 Aug 2016 20:03:48 -0000 Received: from git1-us-west.apache.org (HELO git1-us-west.apache.org) (140.211.11.23) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 02 Aug 2016 20:03:48 +0000 Received: by git1-us-west.apache.org (ASF Mail Server at git1-us-west.apache.org, from userid 33) id 1D323E5CE0; Tue, 2 Aug 2016 20:03:48 +0000 (UTC) Content-Type: text/plain; charset="us-ascii" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit From: kenn@apache.org To: commits@beam.incubator.apache.org Date: Tue, 02 Aug 2016 20:04:27 -0000 Message-Id: <95bb0d3730894f16a845f57895dda871@git.apache.org> In-Reply-To: <4dac88767984489d802aa54016a31f59@git.apache.org> References: <4dac88767984489d802aa54016a31f59@git.apache.org> X-Mailer: ASF-Git Admin Mailer Subject: [41/51] [partial] incubator-beam-site git commit: Regenerate content after merge of #32 archived-at: Tue, 02 Aug 2016 20:04:06 -0000 http://git-wip-us.apache.org/repos/asf/incubator-beam-site/blob/505f6412/content/javadoc/0.1.0-incubating/org/apache/beam/runners/dataflow/DataflowPipelineRegistrar.html ---------------------------------------------------------------------- diff --git a/content/javadoc/0.1.0-incubating/org/apache/beam/runners/dataflow/DataflowPipelineRegistrar.html b/content/javadoc/0.1.0-incubating/org/apache/beam/runners/dataflow/DataflowPipelineRegistrar.html deleted file mode 100644 index fa86b92..0000000 --- a/content/javadoc/0.1.0-incubating/org/apache/beam/runners/dataflow/DataflowPipelineRegistrar.html +++ /dev/null @@ -1,228 +0,0 @@ - - - - - - -DataflowPipelineRegistrar (Apache Beam SDK for Java 0.1.0-incubating API) - - - - - - - - -
- - -
Skip navigation links
- - - - -
-
- - - -
- -
-
-
    -
  • Summary: 
    • -
  • Nested | 
    • -
  • Field | 
    • -
  • Constr | 
    • -
  • Method
    • -
-
    -
  • Detail: 
    • -
  • Field | 
    • -
  • Constr | 
    • -
  • Method
    • -
-
- - -
- - -
-
org.apache.beam.runners.dataflow
-

Class DataflowPipelineRegistrar

-
-
-
    -
  • java.lang.Object
    • -
  • -
      -
    • org.apache.beam.runners.dataflow.DataflowPipelineRegistrar
      • -
    -
    • -
-
- -
-
- -
-
- - -
- - -
Skip navigation links
- - - - -
-
- - - -
- -
-
-
    -
  • Summary: 
    • -
  • Nested | 
    • -
  • Field | 
    • -
  • Constr | 
    • -
  • Method
    • -
-
    -
  • Detail: 
    • -
  • Field | 
    • -
  • Constr | 
    • -
  • Method
    • -
-
- - -
- -

Copyright © 2016 The Apache Software Foundation. All rights reserved.

- - http://git-wip-us.apache.org/repos/asf/incubator-beam-site/blob/505f6412/content/javadoc/0.1.0-incubating/org/apache/beam/runners/dataflow/DataflowPipelineRunner.html ---------------------------------------------------------------------- diff --git a/content/javadoc/0.1.0-incubating/org/apache/beam/runners/dataflow/DataflowPipelineRunner.html b/content/javadoc/0.1.0-incubating/org/apache/beam/runners/dataflow/DataflowPipelineRunner.html deleted file mode 100644 index ff97449..0000000 --- a/content/javadoc/0.1.0-incubating/org/apache/beam/runners/dataflow/DataflowPipelineRunner.html +++ /dev/null @@ -1,504 +0,0 @@ - - - - - - -DataflowPipelineRunner (Apache Beam SDK for Java 0.1.0-incubating API) - - - - - - - - -
- - -
Skip navigation links
- - - - -
-
- - - -
- -
-
- - -
- - -
- - -
-
org.apache.beam.runners.dataflow
-

Class DataflowPipelineRunner

-
-
- -
-
    -
  • -
    -
    -
    public class DataflowPipelineRunner
-extends PipelineRunner<DataflowPipelineJob>
    -
    A PipelineRunner that executes the operations in the - pipeline by first translating them to the Dataflow representation - using the DataflowPipelineTranslator and then submitting - them to a Dataflow service for execution. - -

    Permissions

    - When reading from a Dataflow source or writing to a Dataflow sink using - DataflowPipelineRunner, the Google cloudservices account and the Google compute engine - service account of the GCP project running the Dataflow Job will need access to the corresponding - source/sink. - -

    Please see Google Cloud - Dataflow Security and Permissions for more details.

    -
    • -
-
-
- -
-
-
    -
  • - -
      -
    • - - -

      Field Detail

      - - - -
        -
      • -

        BATCH_WORKER_HARNESS_CONTAINER_IMAGE

        -
        public static final String BATCH_WORKER_HARNESS_CONTAINER_IMAGE
        -
        -
        See Also:
        -
        Constant Field Values
        -
        -
        • -
      - - - -
        -
      • -

        STREAMING_WORKER_HARNESS_CONTAINER_IMAGE

        -
        public static final String STREAMING_WORKER_HARNESS_CONTAINER_IMAGE
        -
        -
        See Also:
        -
        Constant Field Values
        -
        -
        • -
      - - - -
        -
      • -

        PROJECT_ID_REGEXP

        -
        public static final String PROJECT_ID_REGEXP
        -
        Project IDs must contain lowercase letters, digits, or dashes. - IDs must start with a letter and may not end with a dash. - This regex isn't exact - this allows for patterns that would be rejected by - the service, but this is sufficient for basic validation of project IDs.
        -
        -
        See Also:
        -
        Constant Field Values
        -
        -
        • -
      -
      • -
    - - - -
      -
    • - - -

      Method Detail

      - - - -
        -
      • -

        fromOptions

        -
        public static DataflowPipelineRunner fromOptions(PipelineOptions options)
        -
        Construct a runner from the provided options.
        -
        -
        Parameters:
        -
        options - Properties that configure the runner.
        -
        Returns:
        -
        The newly created runner.
        -
        -
        • -
      - - - - - -
        -
      • -

        apply

        -
        public <OutputT extends POutput,InputT extends PInput> OutputT apply(PTransform<InputT,OutputT> transform,
-                                                                     InputT input)
        -
        Applies the given transform to the input. For transforms with customized definitions - for the Dataflow pipeline runner, the application is intercepted and modified here.
        -
        -
        Overrides:
        -
        apply in class PipelineRunner<DataflowPipelineJob>
        -
        -
        • -
      - - - - - - - -
        -
      • -

        getTranslator

        -
        public DataflowPipelineTranslator getTranslator()
        -
        Returns the DataflowPipelineTranslator associated with this object.
        -
        • -
      - - - - - - - - - - - -
        -
      • -

        detectClassPathResourcesToStage

        -
        protected static List<String> detectClassPathResourcesToStage(ClassLoader classLoader)
        -
        Attempts to detect all the resources the class loader has access to. This does not recurse - to class loader parents stopping it from pulling in resources from the system class loader.
        -
        -
        Parameters:
        -
        classLoader - The URLClassLoader to use to detect resources to stage.
        -
        Returns:
        -
        A list of absolute paths to the resources the class loader uses.
        -
        Throws:
        -
        IllegalArgumentException - If either the class loader is not a URLClassLoader or one - of the resources the class loader exposes is not a file resource.
        -
        -
        • -
      -
      • -
    -
    • -
-
-
- - -
- - -
Skip navigation links
- - - - -
-
- - - -
- -
-
- - -
- - -
- -

Copyright © 2016 The Apache Software Foundation. All rights reserved.

- - http://git-wip-us.apache.org/repos/asf/incubator-beam-site/blob/505f6412/content/javadoc/0.1.0-incubating/org/apache/beam/runners/dataflow/DataflowPipelineRunnerHooks.html ---------------------------------------------------------------------- diff --git a/content/javadoc/0.1.0-incubating/org/apache/beam/runners/dataflow/DataflowPipelineRunnerHooks.html b/content/javadoc/0.1.0-incubating/org/apache/beam/runners/dataflow/DataflowPipelineRunnerHooks.html deleted file mode 100644 index acc2fae..0000000 --- a/content/javadoc/0.1.0-incubating/org/apache/beam/runners/dataflow/DataflowPipelineRunnerHooks.html +++ /dev/null @@ -1,287 +0,0 @@ - - - - - - -DataflowPipelineRunnerHooks (Apache Beam SDK for Java 0.1.0-incubating API) - - - - - - - - -
- - -
Skip navigation links
- - - - -
-
- - - -
- -
-
-
    -
  • Summary: 
    • -
  • Nested | 
    • -
  • Field | 
    • -
  • Constr | 
    • -
  • Method
    • -
- -
- - -
- - -
-
org.apache.beam.runners.dataflow
-

Class DataflowPipelineRunnerHooks

-
-
-
    -
  • java.lang.Object
    • -
  • -
      -
    • org.apache.beam.runners.dataflow.DataflowPipelineRunnerHooks
      • -
    -
    • -
-
-
    -
  • -
    -
    -
    @Experimental
-public class DataflowPipelineRunnerHooks
-extends Object
    -
    An instance of this class can be passed to the - DataflowPipelineRunner to add user defined hooks to be - invoked at various times during pipeline execution.
    -
    • -
-
-
- -
-
-
    -
  • - -
      -
    • - - -

      Constructor Detail

      - - - -
        -
      • -

        DataflowPipelineRunnerHooks

        -
        public DataflowPipelineRunnerHooks()
        -
        • -
      -
      • -
    - -
      -
    • - - -

      Method Detail

      - - - -
        -
      • -

        modifyEnvironmentBeforeSubmission

        -
        public void modifyEnvironmentBeforeSubmission(com.google.api.services.dataflow.model.Environment environment)
        -
        Allows the user to modify the environment of their job before their job is submitted - to the service for execution.
        -
        -
        Parameters:
        -
        environment - The environment of the job. Users can make change to this instance in order - to change the environment with which their job executes on the service.
        -
        -
        • -
      -
      • -
    -
    • -
-
-
- - -
- - -
Skip navigation links
- - - - -
-
- - - -
- -
-
-
    -
  • Summary: 
    • -
  • Nested | 
    • -
  • Field | 
    • -
  • Constr | 
    • -
  • Method
    • -
- -
- - -
- -

Copyright © 2016 The Apache Software Foundation. All rights reserved.

- - http://git-wip-us.apache.org/repos/asf/incubator-beam-site/blob/505f6412/content/javadoc/0.1.0-incubating/org/apache/beam/runners/dataflow/DataflowPipelineTranslator.JobSpecification.html ---------------------------------------------------------------------- diff --git a/content/javadoc/0.1.0-incubating/org/apache/beam/runners/dataflow/DataflowPipelineTranslator.JobSpecification.html b/content/javadoc/0.1.0-incubating/org/apache/beam/runners/dataflow/DataflowPipelineTranslator.JobSpecification.html deleted file mode 100644 index 1e6770b..0000000 --- a/content/javadoc/0.1.0-incubating/org/apache/beam/runners/dataflow/DataflowPipelineTranslator.JobSpecification.html +++ /dev/null @@ -1,301 +0,0 @@ - - - - - - -DataflowPipelineTranslator.JobSpecification (Apache Beam SDK for Java 0.1.0-incubating API) - - - - - - - - -
- - -
Skip navigation links
- - - - -
-
- - - -
- -
-
-
    -
  • Summary: 
    • -
  • Nested | 
    • -
  • Field | 
    • -
  • Constr | 
    • -
  • Method
    • -
- -
- - -
- - -
-
org.apache.beam.runners.dataflow
-

Class DataflowPipelineTranslator.JobSpecification

-
-
-
    -
  • java.lang.Object
    • -
  • -
      -
    • org.apache.beam.runners.dataflow.DataflowPipelineTranslator.JobSpecification
      • -
    -
    • -
-
-
    -
  • -
    -
    Enclosing class:
    -
    DataflowPipelineTranslator
    -
    -
    -
    -
    public static class DataflowPipelineTranslator.JobSpecification
-extends Object
    -
    The result of a job translation. - -

    Used to pass the result Job and any state that was used to construct the job that - may be of use to other classes (eg the PTransform to StepName mapping).

    -
    • -
-
-
- -
-
-
    -
  • - -
      -
    • - - -

      Constructor Detail

      - - - -
        -
      • -

        JobSpecification

        -
        public JobSpecification(com.google.api.services.dataflow.model.Job job,
-                        Map<AppliedPTransform<?,?,?>,String> stepNames)
        -
        • -
      -
      • -
    - -
      -
    • - - -

      Method Detail

      - - - -
        -
      • -

        getJob

        -
        public com.google.api.services.dataflow.model.Job getJob()
        -
        • -
      - - - - -
      • -
    -
    • -
-
-
- - -
- - -
Skip navigation links
- - - - -
-
- - - -
- -
-
-
    -
  • Summary: 
    • -
  • Nested | 
    • -
  • Field | 
    • -
  • Constr | 
    • -
  • Method
    • -
- -
- - -
- -

Copyright © 2016 The Apache Software Foundation. All rights reserved.

- - http://git-wip-us.apache.org/repos/asf/incubator-beam-site/blob/505f6412/content/javadoc/0.1.0-incubating/org/apache/beam/runners/dataflow/DataflowPipelineTranslator.TransformTranslator.html ---------------------------------------------------------------------- diff --git a/content/javadoc/0.1.0-incubating/org/apache/beam/runners/dataflow/DataflowPipelineTranslator.TransformTranslator.html b/content/javadoc/0.1.0-incubating/org/apache/beam/runners/dataflow/DataflowPipelineTranslator.TransformTranslator.html deleted file mode 100644 index b26654a..0000000 --- a/content/javadoc/0.1.0-incubating/org/apache/beam/runners/dataflow/DataflowPipelineTranslator.TransformTranslator.html +++ /dev/null @@ -1,235 +0,0 @@ - - - - - - -DataflowPipelineTranslator.TransformTranslator (Apache Beam SDK for Java 0.1.0-incubating API) - - - - - - - - -
- - -
Skip navigation links
- - - - -
-
- - - -
- -
-
-
    -
  • Summary: 
    • -
  • Nested | 
    • -
  • Field | 
    • -
  • Constr | 
    • -
  • Method
    • -
-
    -
  • Detail: 
    • -
  • Field | 
    • -
  • Constr | 
    • -
  • Method
    • -
-
- - -
- - -
-
org.apache.beam.runners.dataflow
-

Interface DataflowPipelineTranslator.TransformTranslator<TransformT extends PTransform>

-
-
-
- -
-
- -
-
- -
-
- - -
- - -
Skip navigation links
- - - - -
-
- - - -
- -
-
-
    -
  • Summary: 
    • -
  • Nested | 
    • -
  • Field | 
    • -
  • Constr | 
    • -
  • Method
    • -
-
    -
  • Detail: 
    • -
  • Field | 
    • -
  • Constr | 
    • -
  • Method
    • -
-
- - -
- -

Copyright © 2016 The Apache Software Foundation. All rights reserved.

- -