beam-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Jingsong Lee (JIRA)" <j...@apache.org>
Subject [jira] [Comment Edited] (BEAM-1393) Update Flink Runner to Flink 1.2.0
Date Sat, 11 Feb 2017 09:08:41 GMT

    [ https://issues.apache.org/jira/browse/BEAM-1393?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15862325#comment-15862325
] 

Jingsong Lee edited comment on BEAM-1393 at 2/11/17 9:08 AM:
-------------------------------------------------------------

BTW, I think SPLIT_DISTRIBUTE state maybe need a new Repartitioner, not only round-robin.
Let each element in ListState have the opportunity to select a KeyGroupIndex. {{CheckpointCoordinator}}
use the KeyGroupIndex to redistribute state. 
I understood it. {{CheckpointCoordinator}} is run in JobManager. That is too heavy to read
every elements.
Maybe Flink can abstract KeyGroup state, provide split(snapshot to several KeyGroups) and
merge(restore by several KeyGroups) methods to manage state with KeyGroups.


was (Author: lzljs3620320):
BTW, I think SPLIT_DISTRIBUTE state maybe need a new Repartitioner, not only round-robin.
Let each element in ListState have the opportunity to select a KeyGroupIndex. {{CheckpointCoordinator}}
use the KeyGroupIndex to redistribute state. 

> Update Flink Runner to Flink 1.2.0
> ----------------------------------
>
>                 Key: BEAM-1393
>                 URL: https://issues.apache.org/jira/browse/BEAM-1393
>             Project: Beam
>          Issue Type: Improvement
>          Components: runner-flink
>            Reporter: Aljoscha Krettek
>            Assignee: Jingsong Lee
>
> When we update to 1.2.0 we can use the new internal Timer API that is available to Flink
operators: {{InternalTimerService}} and also use broadcast state to store side-input data.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

Mime
View raw message