beam-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Kenneth Knowles (JIRA)" <j...@apache.org>
Subject [jira] [Comment Edited] (BEAM-1314) DoFn per-key lifecycle (Setup/Teardown)
Date Wed, 14 Jun 2017 00:05:00 GMT

    [ https://issues.apache.org/jira/browse/BEAM-1314?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15949513#comment-15949513
] 

Kenneth Knowles edited comment on BEAM-1314 at 6/14/17 12:04 AM:
-----------------------------------------------------------------

I think this feature request might make sense, but the description seems a bit wrong. a DoFn
instance that uses state and/or timers can be used for multiple keys, just not concurrently
(because it can't be used concurrently at all).


was (Author: kenn):
I think this feature request might make sense, but the description seems a bit wrong. a DoFn
instance that uses state and/or timers can be used for multiple keys, just not concurrently.

> DoFn per-key lifecycle (Setup/Teardown)
> ---------------------------------------
>
>                 Key: BEAM-1314
>                 URL: https://issues.apache.org/jira/browse/BEAM-1314
>             Project: Beam
>          Issue Type: Wish
>          Components: beam-model
>            Reporter: Eugene Kirpichov
>
> DoFn's that use state and timers are implicitly per-key. Setup/Teardown methods are usually
used to establish expensive resources - long-standing connections and such.
> For per-key DoFn's, we'd often want to use these per-key, so it'd be good to have ability
in the model to ask that there be 1 instance of the DoFn per key, reused between e.g. different
timer or trigger firings for this key, but not used for other keys.
> E.g. this would be particularly useful for Splittable DoFn - for its ability to reuse
expensive resources between checkpoints.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Mime
View raw message