beam-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "ASF GitHub Bot (JIRA)" <>
Subject [jira] [Commented] (BEAM-802) Support Dynamic PipelineOptions for python
Date Wed, 08 Feb 2017 05:28:42 GMT


ASF GitHub Bot commented on BEAM-802:

GitHub user mariapython opened a pull request:

    [BEAM-802] Add ValueProvider class for FileBasedSource I/O Transforms

    - [x] Add ValueProvider class.
    - [x] Derive StaticValueProvider and RuntimeValueProvider from ValueProvider.
    - [x] Derive ValueProviderArgumentParser from argparse.ArgumentParser as API for the template
    - [x] Modify FileBasedSource I/O transforms to accept objects of type ValueProvider.
    - [x] Modify display_data.
    - [x] Handle serialization / deserialization.

You can merge this pull request into a Git repository by running:

    $ git pull ppp_inmaster

Alternatively you can review and apply these changes as the patch at:

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

    This closes #1945
commit 3c04d39d8badcd841894fa6fef78b1888395e273
Author: Maria Garcia Herrero <>
Date:   2017-01-18T06:57:22Z

    Add RuntimeValueProvider class

commit 0636f3cfa8d299e605aeb00f6a6f3d5a000d6877
Author: Maria Garcia Herrero <>
Date:   2017-01-27T19:24:01Z

    Modify IO tests to accept StaticValueProvider objects.

commit cac9b3410a0ba8469b084d606d88ee60c7a09ba9
Author: Maria Garcia Herrero <>
Date:   2017-01-27T21:55:31Z

    Add module to test ValueProvider

commit dd85305d4d4d67a6c7ba271a366a64881cd5c88e
Author: Maria Garcia Herrero <>
Date:   2017-01-27T22:00:15Z

    Fix bug + clean code

commit 3600f8dcd0ceddf43db0f65c3e5af771189766df
Author: Maria Garcia Herrero <>
Date:   2017-01-27T22:02:34Z

    Add serialization for StaticValueProvider

commit cdc53bb479bec67474a511646e778f8a74ccb376
Author: Maria Garcia Herrero <>
Date:   2017-01-28T18:42:11Z

    Remove unnecessary optionsid from pipeline_options and pipeline_options_test

commit f817990537437e7fa7d22031171bf99cf519078f
Author: Maria Garcia Herrero <>
Date:   2017-01-28T18:46:31Z

    Fix bug and correct typos in

commit 169b7ec58fb188db6915da1428323ee014b1ddf6
Author: Maria Garcia Herrero <>
Date:   2017-01-28T21:18:20Z

    Disable validation for RuntimeValueProviders

commit 3d420dd40b7bee69bf7cdcf558c59735d50e2fb5
Author: Maria Garcia Herrero <>
Date:   2017-01-28T21:19:09Z

    Add test to have FileBasedSource accept only str or ValueProvider

commit df0ac1f93703e758d502bbcfb634bc98560b732f
Author: Maria Garcia Herrero <>
Date:   2017-02-06T19:57:40Z

    Have FileSink accept ValueProvider

commit f0dab8f9006177139b906b9cfed808f015cf6c8d
Author: Maria Garcia Herrero <>
Date:   2017-02-08T05:19:05Z

    Fix errors introduced from python-sdk --> master merge


> Support Dynamic PipelineOptions for python
> ------------------------------------------
>                 Key: BEAM-802
>                 URL:
>             Project: Beam
>          Issue Type: New Feature
>          Components: sdk-py
>            Reporter: María GH
>            Assignee: María GH
>            Priority: Minor
>   Original Estimate: 1,680h
>  Remaining Estimate: 1,680h
> Goal:  Enable users to run pipelines from templates filled via CL (pipeline options)
> Background: Currently, the Runner creates the JSON pipeline description which can be
sent to the worker as is, since everything is already defined there (with links to gs:// for
input and binaries). With the parametrized approach, those descriptions are empty and filled
by the user or defaulted, so the pipeline needs to be stored somewhere first until the values
become available.
> Tasks:
> 1- Create template-style pipeline description (TemplateRunner)
> The graph description is now a template (some parts are not filled) that needs to be
> 2- Define values to inject to the template (ValueProviders API)
> The placeholders can be filled with default values (static) or with dynamic key/value
pairs provided at runtime (dynamic)

This message was sent by Atlassian JIRA

View raw message