flink-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Stefan Richter (JIRA)" <j...@apache.org>
Subject [jira] [Created] (FLINK-5051) Backwards compatibility for serializers in backend state
Date Fri, 11 Nov 2016 09:53:58 GMT
Stefan Richter created FLINK-5051:

             Summary: Backwards compatibility for serializers in backend state
                 Key: FLINK-5051
                 URL: https://issues.apache.org/jira/browse/FLINK-5051
             Project: Flink
          Issue Type: Improvement
          Components: State Backends, Checkpointing
            Reporter: Stefan Richter

When a new state is register, e.g. in a keyed backend via `getPartitionedState`, the caller
has to provide all type serializers required for the persistence of state components. Explicitly
passing the serializers on state creation already allows for potentiall version upgrades of

However, those serializers are currently not part of any snapshot and are only provided at
runtime, when the state is registered newly or restored. For backwards compatibility, this
has strong implications: checkpoints are not self contained in that state is currently a blackbox
without knowledge about it's corresponding serializers. Most cases where we would need to
restructure the state are basically lost. We could only convert them lazily at runtime and
only once the user is registering the concrete state, which might happen at unpredictable

I suggest to adapt our solution as follows:

- As now, all states are registered with their set of serializers.
- Unlike now, all serializers are written to the snapshot. This makes savepoints self-contained
and also allows to create inspection tools for savepoints at some point in the future.
- Introduce an interface {{Versioned}} with {{long getVersion()}} and {{boolean isCompatible(Versioned
v)}} which is then implemented by serializers. Compatible serializers must ensure that they
can deserialize older versions, and can then serialize them in their new format. This is how
we upgrade.

This message was sent by Atlassian JIRA

View raw message