beam-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "ASF GitHub Bot (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (BEAM-115) Beam Runner API
Date Sat, 22 Apr 2017 00:22:04 GMT

    [ https://issues.apache.org/jira/browse/BEAM-115?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15979640#comment-15979640
] 

ASF GitHub Bot commented on BEAM-115:
-------------------------------------

GitHub user robertwb opened a pull request:

    https://github.com/apache/beam/pull/2644

    [BEAM-115] Fn API support for Python

    Be sure to do all of the following to help us incorporate your contribution
    quickly and easily:
    
     - [ ] Make sure the PR title is formatted like:
       `[BEAM-<Jira issue #>] Description of pull request`
     - [ ] Make sure tests pass via `mvn clean verify`. (Even better, enable
           Travis-CI on your fork and ensure the whole test matrix passes).
     - [ ] Replace `<Jira issue #>` in the title with the actual Jira issue
           number, if there is one.
     - [ ] If this contribution is large, please file an Apache
           [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.pdf).
    
    ---


You can merge this pull request into a Git repository by running:

    $ git pull https://github.com/robertwb/incubator-beam fn-api

Alternatively you can review and apply these changes as the patch at:

    https://github.com/apache/beam/pull/2644.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

    This closes #2644
    
----
commit 4f5c5241949473d81b18cb034640189ea98c49df
Author: Robert Bradshaw <robertwb@gmail.com>
Date:   2017-04-21T16:59:52Z

    Add instructions to regenerate Python proto wrappers.

commit f4f94db5bb3b64c015fd95de64bafd34770aaa21
Author: Robert Bradshaw <robertwb@gmail.com>
Date:   2017-04-21T17:00:20Z

    Generate python proto wrappers for runner and fn API.

commit 6f0e487749b777c5606a69865b7e47134d878a58
Author: Robert Bradshaw <robertwb@gmail.com>
Date:   2017-04-21T18:04:03Z

    Ignore generated files for linter.

commit aa1425c86f5def43ff9032b8666352b747d20440
Author: Robert Bradshaw <robertwb@gmail.com>
Date:   2017-04-21T18:09:03Z

    Ignore generated files in rat plugin.

commit 91428b06f8d4ced0bd3965801920cd94ee8298c6
Author: Robert Bradshaw <robertwb@gmail.com>
Date:   2017-04-21T19:15:02Z

    Add apache licence to generated files.

commit 27f718e89d1829b5b217f263d4b4deb8832c947d
Author: Robert Bradshaw <robertwb@gmail.com>
Date:   2017-04-20T20:59:25Z

    Add fn api runner.

commit f987622d524fa4245089f82fbd150c0cf9ae380d
Author: Robert Bradshaw <robertwb@gmail.com>
Date:   2017-04-20T21:20:29Z

    Restore __init__.py

commit 45ccb571aed5e51ef893428c2a5350ba01864aac
Author: Robert Bradshaw <robertwb@robertwb-u.cbf.corp.google.com>
Date:   2017-04-20T20:44:33Z

    Add runner core files.

commit c6310ad8236578c05432244ced40fe572c2d71a7
Author: Robert Bradshaw <robertwb@gmail.com>
Date:   2017-04-20T21:23:30Z

    move files around

commit 0f9de53c7007e200e5f04c9a30349313b2bdff39
Author: Robert Bradshaw <robertwb@gmail.com>
Date:   2017-04-20T21:27:14Z

    more moving

commit 2f5c518bff028878b7d18a30141beea734cb36a0
Author: Robert Bradshaw <robertwb@gmail.com>
Date:   2017-04-20T21:43:28Z

    Rename runners.

commit 85b2172c1d8d6a5797015c15cd5a301de4a252c6
Author: Robert Bradshaw <robertwb@gmail.com>
Date:   2017-04-20T21:53:26Z

    cythonization works

commit 9f1177db48fa0230fab463cb7a5d784b96c9e084
Author: Robert Bradshaw <robertwb@gmail.com>
Date:   2017-04-20T22:03:52Z

    test module import renames

commit a0e8d792bbfac4550e4399b29c9004f67b82cbd0
Author: Robert Bradshaw <robertwb@gmail.com>
Date:   2017-04-20T22:06:30Z

    remove google3s

commit dfc0b2f3ca04594a2167eaadc7882ca257cdca5f
Author: Robert Bradshaw <robertwb@gmail.com>
Date:   2017-04-20T22:07:37Z

    move sdk_harness to sdk_worker

commit 8b08216c54e65fd0eb5a6c9588405cb4301267b2
Author: Robert Bradshaw <robertwb@gmail.com>
Date:   2017-04-20T22:09:33Z

    Remove unused end_time from statesampler.

commit 011dec575a4066eb7e50c0bff2bad78a9cadd449
Author: Robert Bradshaw <robertwb@gmail.com>
Date:   2017-04-21T16:59:52Z

    Add instructions to regenerate Python proto wrappers.

commit 4a74cdd528c6e9800c66152f9498592f51b59248
Author: Robert Bradshaw <robertwb@gmail.com>
Date:   2017-04-21T17:00:20Z

    Generate python proto wrappers for runner and fn API.

commit 5dcd25d144f784a5b9bd6bd668c2ceb8bc2680f3
Author: Robert Bradshaw <robertwb@gmail.com>
Date:   2017-04-21T18:04:03Z

    Ignore generated files for linter.

commit 5697ecdca1c14ce926f0be63399df77609c12aaf
Author: Robert Bradshaw <robertwb@gmail.com>
Date:   2017-04-21T18:09:03Z

    Ignore generated files in rat plugin.

commit 57a16aa810f5e4ef14c5f2ced901c3bb6fbc7b1a
Author: Robert Bradshaw <robertwb@gmail.com>
Date:   2017-04-21T19:15:02Z

    Add apache licence to generated files.

commit a8456be3830869e5b51e24970655819bc832bd79
Author: Robert Bradshaw <robertwb@gmail.com>
Date:   2017-04-21T19:36:27Z

    implement portpicker, fix more imports

commit c4633f174d436304da142606145a4827c9da7f4e
Author: Robert Bradshaw <robertwb@gmail.com>
Date:   2017-04-21T20:04:22Z

    portpicker fixes

commit f349f14e49e03b675ad7f7ab06e2c871c9d5eae4
Author: Robert Bradshaw <robertwb@gmail.com>
Date:   2017-04-21T20:04:35Z

    more import fixes

commit cc1dc88e83a5ed7ba0c04fbd10943ec98d920c73
Author: Robert Bradshaw <robertwb@gmail.com>
Date:   2017-04-21T21:22:07Z

    Adapt to PR #2505 changes to protos.

commit cb09e464c3da6358b203fad681bba41fb3d4a837
Author: Robert Bradshaw <robertwb@gmail.com>
Date:   2017-04-21T21:22:37Z

    Merge branch 'gen-protos' into fn-api

commit a11349156833c9fb1df0263c9289f9bebe1c51d3
Author: Robert Bradshaw <robertwb@gmail.com>
Date:   2017-04-21T21:38:12Z

    disble simple state creation

commit b64991b2f08a5f21da440f86f90870e323ae0407
Author: Robert Bradshaw <robertwb@gmail.com>
Date:   2017-04-21T21:43:28Z

    more package renames

commit c857789993cc8333c7d27c3527701e794fd3615d
Author: Robert Bradshaw <robertwb@gmail.com>
Date:   2017-04-21T21:50:00Z

    remove google3, fix imports

commit acc28a2b75cb338ed6f1f7e9edebec646c1e29db
Author: Robert Bradshaw <robertwb@gmail.com>
Date:   2017-04-21T21:52:25Z

    fix more imports, google3 in portability dir

----


> Beam Runner API
> ---------------
>
>                 Key: BEAM-115
>                 URL: https://issues.apache.org/jira/browse/BEAM-115
>             Project: Beam
>          Issue Type: Improvement
>          Components: beam-model-runner-api
>            Reporter: Kenneth Knowles
>            Assignee: Kenneth Knowles
>
> The PipelineRunner API from the SDK is not ideal for the Beam technical vision.
> It has technical limitations:
>  - The user's DAG (even including library expansions) is never explicitly represented,
so it cannot be analyzed except incrementally, and cannot necessarily be reconstructed (for
example, to display it!).
>  - The flattened DAG of just primitive transforms isn't well-suited for display or transform
override.
>  - The TransformHierarchy isn't well-suited for optimizations.
>  - The user must realistically pre-commit to a runner, and its configuration (batch vs
streaming) prior to graph construction, since the runner will be modifying the graph as it
is built.
>  - It is fairly language- and SDK-specific.
> It has usability issues (these are not from intuition, but derived from actual cases
of failure to use according to the design)
>  - The interleaving of apply() methods in PTransform/Pipeline/PipelineRunner is confusing.
>  - The TransformHierarchy, accessible only via visitor traversals, is cumbersome.
>  - The staging of construction-time vs run-time is not always obvious.
> These are just examples. This ticket tracks designing, coming to consensus, and building
an API that more simply and directly supports the technical vision.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

Mime
View raw message