beam-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "ASF GitHub Bot (JIRA)" <>
Subject [jira] [Commented] (BEAM-757) The SparkRunner should utilize the SDK's DoFnRunner instead of writing it's own.
Date Mon, 12 Dec 2016 09:41:58 GMT


ASF GitHub Bot commented on BEAM-757:

GitHub user amitsela opened a pull request:

    [BEAM-757, BEAM-807]

    Be sure to do all of the following to help us incorporate your contribution
    quickly and easily:
     - [ ] Make sure the PR title is formatted like:
       `[BEAM-<Jira issue #>] Description of pull request`
     - [ ] Make sure tests pass via `mvn clean verify`. (Even better, enable
           Travis-CI on your fork and ensure the whole test matrix passes).
     - [ ] Replace `<Jira issue #>` in the title with the actual Jira issue
           number, if there is one.
     - [ ] If this contribution is large, please file an Apache
           [Individual Contributor License Agreement](

You can merge this pull request into a Git repository by running:

    $ git pull new-do-fn

Alternatively you can review and apply these changes as the patch at:

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

    This closes #1578
commit 5748fe9cfd9c36d92b2870acde5e071ea696ac78
Author: Sela <>
Date:   2016-12-11T12:30:24Z

    Use DoFnRunner in the implementation os FlatMapFunction.
    Migrate to new DoFn.

commit 7f73c991426e0602d318a47cd9a38a3eebd979bf
Author: Sela <>
Date:   2016-12-11T12:31:59Z

    Implement AggregatorFactory for Spark runner, to be used by DoFnRunner.

commit b2b0d463b0cf2105248d02411a190f6406f26c69
Author: Sela <>
Date:   2016-12-11T12:32:49Z

    Migrate to new DoFn.

commit 00219617355e1d77d0bba6c272c0a2b3595eac61
Author: Sela <>
Date:   2016-12-11T19:09:47Z

    Add a custom AssignWindows implementation.

commit cd574d62825d636bb30ab0fd13172d2f8bb5cbb7
Author: Sela <>
Date:   2016-12-12T09:33:58Z

    Setup and teardown DoFn.

commit 9355b1ef4943b29e7d26735484aad2e63bb1d1eb
Author: Sela <>
Date:   2016-12-12T09:34:17Z

    Add implementation for GroupAlsoByWindow via flatMap.


> The SparkRunner should utilize the SDK's DoFnRunner instead of writing it's own.
> --------------------------------------------------------------------------------
>                 Key: BEAM-757
>                 URL:
>             Project: Beam
>          Issue Type: Improvement
>          Components: runner-spark
>            Reporter: Amit Sela
>            Assignee: Amit Sela
> The SDK now provides DoFnRunner implementations, and so to avoid maintaining against
the SDK, the runner should leverage the runner API instead.

This message was sent by Atlassian JIRA

View raw message