Mailing-List: contact commits-help@beam.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@beam.apache.org
Date: Fri, 17 Feb 2017 04:36:41 +0000 (UTC)
From: "ASF GitHub Bot (JIRA)" <jira@apache.org>
To: commits@beam.incubator.apache.org
Message-ID: <JIRA.12950173.1458016224000.106365.1487306201688@Atlassian.JIRA>
In-Reply-To: <JIRA.12950173.1458016224000@Atlassian.JIRA>
References: <JIRA.12950173.1458016224000@Atlassian.JIRA> <JIRA.12950173.1458016224596@jira-lw-us.apache.org>
Subject: [jira] [Commented] (BEAM-115) Beam Runner API
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
archived-at: Fri, 17 Feb 2017 04:36:50 -0000


    [ https://issues.apache.org/jira/browse/BEAM-115?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15871158#comment-15871158 ] 

ASF GitHub Bot commented on BEAM-115:
-------------------------------------

GitHub user kennknowles opened a pull request:

    https://github.com/apache/beam/pull/2030

    [BEAM-115,BEAM-1328] Convert to/from WindowingStrategy proto in Java SDK

    Be sure to do all of the following to help us incorporate your contribution
    quickly and easily:
    
     - [x] Make sure the PR title is formatted like:
       `[BEAM-<Jira issue #>] Description of pull request`
     - [x] Make sure tests pass via `mvn clean verify`. (Even better, enable
           Travis-CI on your fork and ensure the whole test matrix passes).
     - [x] Replace `<Jira issue #>` in the title with the actual Jira issue
           number, if there is one.
     - [x] If this contribution is large, please file an Apache
           [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.txt).
    
    ---
    
    Some aspects of this PR are sort of hacks, but the kind that might be forgiven in order to make rapid progress. I am opening the PR for comment anyhow, but there are  few changes that I might now make underneath this PR:
    
    1. Move Java SDK `ClosingBehavior` to top level and rename. Incidentally related to [BEAM-210](https://issues.apache.org/jira/browse/BEAM-210) only because that is another reason this should just be a top-level concept. Not sure there's much benefit to putting public enums inside other misc classes when there's no real natural home for them.
    2. Move Java SDK `AccumulationMode` to top level and put its to/from proto there. In particular, it should also come out of `util`.
    3. Maybe actually try to move from `OutputTimeFn` to a Java SDK `OutputTime` prior to this PR. But actually converting this to proto, then converting the runners to use that will make the latter migration easier. Some cruft is introduced in the meantime, though.
    4. If `WindowingStrategy` is going to continue to be a thing that is prominent all over our public SDK surface and in the runner API whether it really belongs in `util`.
    
    And I'll want to flesh out the list of test cases. Given how generic the logic is, I don't expect many surprises.
    
    R: @tgroh 

You can merge this pull request into a Git repository by running:

    $ git pull https://github.com/kennknowles/beam WindowingStrategy-from-proto

Alternatively you can review and apply these changes as the patch at:

    https://github.com/apache/beam/pull/2030.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

    This closes #2030
    
----
commit e7171b8fcd8068e5b0ec0c660d16d104bb1c334a
Author: Kenneth Knowles <klk@google.com>
Date:   2017-02-16T22:45:05Z

    Make SDK-specific serialized blob really a blob

commit b3b3ba5cdd5559685208b0cbbd45071bc862a7b8
Author: Kenneth Knowles <klk@google.com>
Date:   2017-02-17T04:26:39Z

    Add closing behavior to Runner API proto

commit fd995928091582487e14cb1af128354b5c9fadbe
Author: Kenneth Knowles <klk@google.com>
Date:   2017-02-17T04:26:45Z

    Add conversion to/from Runner API proto for WindowingStrategy

----


> Beam Runner API
> ---------------
>
>                 Key: BEAM-115
>                 URL: https://issues.apache.org/jira/browse/BEAM-115
>             Project: Beam
>          Issue Type: Improvement
>          Components: beam-model-runner-api
>            Reporter: Kenneth Knowles
>            Assignee: Kenneth Knowles
>
> The PipelineRunner API from the SDK is not ideal for the Beam technical vision.
> It has technical limitations:
>  - The user's DAG (even including library expansions) is never explicitly represented, so it cannot be analyzed except incrementally, and cannot necessarily be reconstructed (for example, to display it!).
>  - The flattened DAG of just primitive transforms isn't well-suited for display or transform override.
>  - The TransformHierarchy isn't well-suited for optimizations.
>  - The user must realistically pre-commit to a runner, and its configuration (batch vs streaming) prior to graph construction, since the runner will be modifying the graph as it is built.
>  - It is fairly language- and SDK-specific.
> It has usability issues (these are not from intuition, but derived from actual cases of failure to use according to the design)
>  - The interleaving of apply() methods in PTransform/Pipeline/PipelineRunner is confusing.
>  - The TransformHierarchy, accessible only via visitor traversals, is cumbersome.
>  - The staging of construction-time vs run-time is not always obvious.
> These are just examples. This ticket tracks designing, coming to consensus, and building an API that more simply and directly supports the technical vision.


--
This message was sent by Atlassian JIRA
(v6.3.15#6346)