flink-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "ASF GitHub Bot (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (FLINK-6014) Allow the registration of state objects in checkpoints
Date Mon, 20 Mar 2017 02:28:41 GMT

    [ https://issues.apache.org/jira/browse/FLINK-6014?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15932078#comment-15932078

ASF GitHub Bot commented on FLINK-6014:

Github user shixiaogang commented on the issue:

    @StephanEwen Thanks a lot for your valuable comments. I will update the PR as suggested.
    * I think it's a good idea that we make the `StateRegistry` into `SharedStateRegistry`.
 That will need (1) the state handle must not discard registered objects in the `discardState`
method and (2) the state handle has to register shared object once it is received by the coordinator
(now the state handle does not register their objects before its checkpoint completes).
    * Now that state handles have to register their objects once they are received by the
coordinator, we should move `SharedStateRegister` from `CompletedCheckpointStore` to `CheckpointCoordinator`.
    * It's better for `StateRegistry` directly discard an object once its reference count
is 0. I used a list to collect discarded objects because I want to make the changes in the
discarding of completed checkpoint as few as possible. 

> Allow the registration of state objects in checkpoints
> ------------------------------------------------------
>                 Key: FLINK-6014
>                 URL: https://issues.apache.org/jira/browse/FLINK-6014
>             Project: Flink
>          Issue Type: Sub-task
>          Components: State Backends, Checkpointing
>            Reporter: Xiaogang Shi
>            Assignee: Xiaogang Shi
> This issue is the very first step towards incremental checkpointing. We introduce a new
state handle named {{CompositeStateHandle}} to be the base of the snapshots taken by task
components.  Known implementation may include {{KeyedStateHandle}} (for {{KeyedStateBackend}}s),
{{SubtaskState}} (for subtasks, splits of {{JobVertex}}) and {{TaskState}} (for {{JobVertex}}s).
> Each {{CompositeStateHandle}} is composed of a collection of {{StateObject}s. It should
register all its state objects in {{StateRegistry}} when its checkpoint is added into {{CompletedCheckpointStore}}
(i.e., a pending checkpoint completes or a complete checkpoint is reloaded in the recovery).

> When a completed checkpoint is moved out of the {{CompletedCheckpointStore}}, we should
not simply discard all state objects in the checkpoint. With the introduction of incremental
checkpointing, a {{StateObject}} may be referenced by different checkpoints. We should unregister
all the state objects contained in the {{StateRegistry}} first. Only those state objects that
are not referenced by any checkpoint can be deleted.

This message was sent by Atlassian JIRA

View raw message