beam-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "ASF GitHub Bot (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (BEAM-65) SplittableDoFn
Date Thu, 02 Feb 2017 00:09:51 GMT

    [ https://issues.apache.org/jira/browse/BEAM-65?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15849152#comment-15849152
] 

ASF GitHub Bot commented on BEAM-65:
------------------------------------

GitHub user jkff opened a pull request:

    https://github.com/apache/beam/pull/1895

    [BEAM-65] ProcessFn: create invoker in setup()

    This makes more sense than creating it in constructor, since
    the fn can be serialized (and will be - by Dataflow runner).
    
    However, since direct runner creates the fn explicitly during
    evaluation, we need to also call setup() explicitly.
    
    R: @kennknowles 

You can merge this pull request into a Git repository by running:

    $ git pull https://github.com/jkff/incubator-beam sdf-setup

Alternatively you can review and apply these changes as the patch at:

    https://github.com/apache/beam/pull/1895.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

    This closes #1895
    
----
commit 5546ad68fd18315a251d059b7c8c512640172d24
Author: Eugene Kirpichov <kirpichov@google.com>
Date:   2017-02-02T00:07:44Z

    [BEAM-65] ProcessFn: create invoker in setup()
    
    This makes more sense than creating it in constructor, since
    the fn can be serialized (and will be - by Dataflow runner).
    
    However, since direct runner creates the fn explicitly during
    evaluation, we need to also call setup() explicitly.

----


> SplittableDoFn
> --------------
>
>                 Key: BEAM-65
>                 URL: https://issues.apache.org/jira/browse/BEAM-65
>             Project: Beam
>          Issue Type: New Feature
>          Components: beam-model
>            Reporter: Daniel Halperin
>            Assignee: Eugene Kirpichov
>            Priority: Minor
>
> SplittableDoFn is a proposed enhancement for "dynamically splittable work" to the Beam
model.
> Among other things, it would allow a unified implementation of bounded/unbounded sources
with dynamic work rebalancing and the ability to express multiple scalable steps (e.g., global
expansion -> file sizing & parsing -> splitting files into independently-processable
blocks) via composition rather than inheritance.
> This would make it much easier to implement many types of sources, to modify and reuse
existing sources. Also, it would improve scalability of the Beam model by moving things like
splitting a source from the control plane (where it is today -- glob -> List<FileBasedSource>
sent over service APIs) into the data plane (PCollection<Glob> -> PCollection<FileName>
-> ...).



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

Mime
View raw message