Mailing-List: contact commits-help@beam.incubator.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@beam.incubator.apache.org
Date: Thu, 10 Nov 2016 22:11:58 +0000 (UTC)
From: "ASF GitHub Bot (JIRA)" <jira@apache.org>
To: commits@beam.incubator.apache.org
Message-ID: <JIRA.13014627.1477282526000.248757.1478815918377@Atlassian.JIRA>
In-Reply-To: <JIRA.13014627.1477282526000@Atlassian.JIRA>
References: <JIRA.13014627.1477282526000@Atlassian.JIRA> <JIRA.13014627.1477282526016@arcas>
Subject: [jira] [Commented] (BEAM-802) Support Dynamic PipelineOptions for
 python
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: quoted-printable
archived-at: Thu, 10 Nov 2016 22:12:04 -0000


    [ https://issues.apache.org/jira/browse/BEAM-802?page=3Dcom.atlassian.j=
ira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=3D156553=
14#comment-15655314 ]=20

ASF GitHub Bot commented on BEAM-802:
-------------------------------------

GitHub user mariapython opened a pull request:

    https://github.com/apache/incubator-beam/pull/1342

    [BEAM-802] -- Add ability to stage a job and/or stop the execution

    Modify create_job to allow staging the job and not submitting it to the=
 service.
    - Modularize create_job in create job description, stage job, and send =
for execution.
    - Modify --dataflow_job_file to stage the job and continue submitting i=
t to the service.
    - Add --template_location to stage the job but not submit it to the ser=
vice.
    - Add tests for both, including making them mutually exclusive (followi=
ng Java SDK decision).
    - Add template_runner_test.py with integration tests.

You can merge this pull request into a Git repository by running:

    $ git pull https://github.com/mariapython/incubator-beam python-sdk

Alternatively you can review and apply these changes as the patch at:

    https://github.com/apache/incubator-beam/pull/1342.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

    This closes #1342
   =20
----
commit 3b46dbc7729bd4a3c83e2aa985c7e20b8bc2210b
Author: Maria Garcia Herrero <mariagh@google.com>
Date:   2016-11-02T16:14:48Z

    Modify create_job to allow staging the job and not submitting it to the=
 service.
   =20
    - Modularize create_job in create job description, stage job, and send =
for execution.
    - Modify --dataflow_job_file to stage the job and continue submitting i=
t to the service.
    - Add --template_location to stage the job but not submit it to the ser=
vice.
    - Add tests for both, including making them mutually exclusive (followi=
ng Java SDK decision).
    - Add template_runner_test.py with integration tests.

----


> Support Dynamic PipelineOptions for python
> ------------------------------------------
>
>                 Key: BEAM-802
>                 URL: https://issues.apache.org/jira/browse/BEAM-802
>             Project: Beam
>          Issue Type: New Feature
>          Components: sdk-py
>            Reporter: Mar=C3=ADa GH
>            Assignee: Mar=C3=ADa GH
>            Priority: Minor
>   Original Estimate: 1,680h
>  Remaining Estimate: 1,680h
>
> Goal:  Enable users to run pipelines from templates filled via CL (pipeli=
ne options)
> Background: Currently, the Runner creates the JSON pipeline description w=
hich can be sent to the worker as is, since everything is already defined t=
here (with links to gs:// for input and binaries). With the parametrized ap=
proach, those descriptions are empty and filled by the user or defaulted, s=
o the pipeline needs to be stored somewhere first until the values become a=
vailable.
> Tasks:
> 1- Create template-style pipeline description (TemplateRunner)
> The graph description is now a template (some parts are not filled) that =
needs to be saved.
> 2- Define values to inject to the template (ValueProviders API)
> The placeholders can be filled with default values (static) or with dynam=
ic key/value pairs provided at runtime (dynamic)


--
This message was sent by Atlassian JIRA
(v6.3.4#6332)