beam-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "ASF GitHub Bot (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (BEAM-802) Support Dynamic PipelineOptions for python
Date Thu, 10 Nov 2016 22:11:58 GMT

    [ https://issues.apache.org/jira/browse/BEAM-802?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15655314#comment-15655314
] 

ASF GitHub Bot commented on BEAM-802:
-------------------------------------

GitHub user mariapython opened a pull request:

    https://github.com/apache/incubator-beam/pull/1342

    [BEAM-802] -- Add ability to stage a job and/or stop the execution

    Modify create_job to allow staging the job and not submitting it to the service.
    - Modularize create_job in create job description, stage job, and send for execution.
    - Modify --dataflow_job_file to stage the job and continue submitting it to the service.
    - Add --template_location to stage the job but not submit it to the service.
    - Add tests for both, including making them mutually exclusive (following Java SDK decision).
    - Add template_runner_test.py with integration tests.

You can merge this pull request into a Git repository by running:

    $ git pull https://github.com/mariapython/incubator-beam python-sdk

Alternatively you can review and apply these changes as the patch at:

    https://github.com/apache/incubator-beam/pull/1342.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

    This closes #1342
    
----
commit 3b46dbc7729bd4a3c83e2aa985c7e20b8bc2210b
Author: Maria Garcia Herrero <mariagh@google.com>
Date:   2016-11-02T16:14:48Z

    Modify create_job to allow staging the job and not submitting it to the service.
    
    - Modularize create_job in create job description, stage job, and send for execution.
    - Modify --dataflow_job_file to stage the job and continue submitting it to the service.
    - Add --template_location to stage the job but not submit it to the service.
    - Add tests for both, including making them mutually exclusive (following Java SDK decision).
    - Add template_runner_test.py with integration tests.

----


> Support Dynamic PipelineOptions for python
> ------------------------------------------
>
>                 Key: BEAM-802
>                 URL: https://issues.apache.org/jira/browse/BEAM-802
>             Project: Beam
>          Issue Type: New Feature
>          Components: sdk-py
>            Reporter: María GH
>            Assignee: María GH
>            Priority: Minor
>   Original Estimate: 1,680h
>  Remaining Estimate: 1,680h
>
> Goal:  Enable users to run pipelines from templates filled via CL (pipeline options)
> Background: Currently, the Runner creates the JSON pipeline description which can be
sent to the worker as is, since everything is already defined there (with links to gs:// for
input and binaries). With the parametrized approach, those descriptions are empty and filled
by the user or defaulted, so the pipeline needs to be stored somewhere first until the values
become available.
> Tasks:
> 1- Create template-style pipeline description (TemplateRunner)
> The graph description is now a template (some parts are not filled) that needs to be
saved.
> 2- Define values to inject to the template (ValueProviders API)
> The placeholders can be filled with default values (static) or with dynamic key/value
pairs provided at runtime (dynamic)



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message