flink-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Aljoscha Krettek (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (FLINK-6383) Add versioning for AbstractStreamOperator state
Date Tue, 25 Apr 2017 16:24:05 GMT

    [ https://issues.apache.org/jira/browse/FLINK-6383?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15983153#comment-15983153

Aljoscha Krettek commented on FLINK-6383:

I had a quick offline chat with [~plucas] about this and it seemed like now would be a good
time to add this. We could also introduce the whole scheme the first time that we want to
break the scheme, it will work because of the magic number.

[~StephanEwen], [~srichter] what do you think?

> Add versioning for AbstractStreamOperator state
> -----------------------------------------------
>                 Key: FLINK-6383
>                 URL: https://issues.apache.org/jira/browse/FLINK-6383
>             Project: Flink
>          Issue Type: Improvement
>          Components: DataStream API
>    Affects Versions: 1.3.0
>            Reporter: Aljoscha Krettek
>            Assignee: Aljoscha Krettek
>            Priority: Blocker
>             Fix For: 1.3.0
> Right now, {{AbstractStreamOperator}} checkpoints key-group state in {{AbstractStreamOperator.snapshotState(StateSnapshotContext
context)}}. Currently, this is the timers for each key group. We might want to change the
format of this or what we store in the future. In order to facilitate this we should write
a version to the beginning of the stream so that we can deal with schema updates.
> For dealing with the fact that we don't currently write a version number we have to at
a magic number add the beginning (think {{0xCAFEBABE}}). For dealing with state that was written
prior to versioning we read the first 8 bytes, buffer them, if they match the magic number
we read the version and from then on jump to the logic based on the version. If the magic
number doesn't match we know we have state from a pre-version version and read the stream
using the old code, including the 8 buffered bytes.

This message was sent by Atlassian JIRA

View raw message