flink-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Tzu-Li (Gordon) Tai (JIRA)" <j...@apache.org>
Subject [jira] [Resolved] (FLINK-6804) Inconsistent state migration behaviour between different state backends
Date Tue, 13 Jun 2017 05:56:00 GMT

     [ https://issues.apache.org/jira/browse/FLINK-6804?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel

Tzu-Li (Gordon) Tai resolved FLINK-6804.
       Resolution: Fixed
    Fix Version/s: 1.4.0

Fixed for 1.31. via 379be13b67948d28be66e071072412c870d6e1f8.
Fixed for master via f0f2e99b6c829c4f4e2ca47c7647a64fe0c9d808.

> Inconsistent state migration behaviour between different state backends
> -----------------------------------------------------------------------
>                 Key: FLINK-6804
>                 URL: https://issues.apache.org/jira/browse/FLINK-6804
>             Project: Flink
>          Issue Type: Bug
>          Components: State Backends, Checkpointing, Type Serialization System
>    Affects Versions: 1.3.0, 1.4.0
>            Reporter: Till Rohrmann
>            Assignee: Tzu-Li (Gordon) Tai
>            Priority: Critical
>              Labels: flink-rel-1.3.1-blockers
>             Fix For: 1.3.1, 1.4.0
> The {{MemoryStateBackend}}, {{FsStateBackend}} and {{RocksDBStateBackend}} show a different
behaviour when it comes to recovery from old state and state migration. For example, using
the {{MemoryStateBackend}} it is possible to recover pojos which now have additional fields
(at recovery time). The only caveat is that the recovered {{PojoSerializer}} will silently
drop the added fields when writing the new Pojo. In contrast, the {{RocksDBStateBackend}}
correctly recognizes that a state migration is necessary and thus fails with a {{StateMigrationException}}.
The same applies to the case where Pojo field types change. The {{MemoryStateBackend}} and
the {{FsStateBackend}} accept such a change as long as the fields still have the same length.
The {{RocksDBStateBackend}} correctly fails with a {{StateMigrationException}}.
> I think that all state backends should behave similarly and give the user the same recovery
and state migration guarantees. Otherwise, it could happen that jobs run with one state backend
but not with another (wrt semantic behaviour).

This message was sent by Atlassian JIRA

View raw message