flink-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Tzu-Li (Gordon) Tai (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (FLINK-6425) Integrate serializer reconfiguration into state restore flow to activate serializer upgrades
Date Sun, 07 May 2017 19:50:04 GMT

     [ https://issues.apache.org/jira/browse/FLINK-6425?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel

Tzu-Li (Gordon) Tai updated FLINK-6425:
    Fix Version/s: 1.3.0

> Integrate serializer reconfiguration into state restore flow to activate serializer upgrades
> --------------------------------------------------------------------------------------------
>                 Key: FLINK-6425
>                 URL: https://issues.apache.org/jira/browse/FLINK-6425
>             Project: Flink
>          Issue Type: Sub-task
>          Components: State Backends, Checkpointing
>            Reporter: Tzu-Li (Gordon) Tai
>            Assignee: Tzu-Li (Gordon) Tai
>             Fix For: 1.3.0
> With FLINK-6191, {{TypeSerializer}} will be reconfigurable.
> From the state backends' point of view, serializer reconfiguration doubles as a mechanism
to determine how serializer upgrades should be handled.
> The general idea is that state checkpoints should contain the following as the state's
> - the previous serializer
> - snapshot of the previous serializer's configuration
> The upgrade flow is as follows:
> 1. On restore, try to deserialize the previous old serializer. Deserialization may fail
if a) the serializer no longer exists in classpath, or b) the serializer class is not longer
valid (i.e., implementation changed and resulted in different serialVersionUID). In this case,
use a dummy serializer as a placeholder. This dummy serializer is currently the {{ClassNotFoundProxySerializer}}
in the code.
> 2. Deserialize the configuration snapshot of the previous old serializer. The configuration
snapshot must be successfully deserialized, otherwise the state restore fails.
> 3. When we get the new registered serializer for the state (could be a completely new
serializer, the same serializer with different implementations, or the exact same serializer
untouched; either way they are seen as a new serializer), we use the configuration snapshot
of the old serializer to reconfigure the new serializer.
> This completes the upgrade of the old serializer.  However, depending on the result of
the upgrade, state conversion needs to take place (for now, if state conversion is required,
we just fail the job as this functionality isn't available yet). The results could be:
> - Compatible: restore success + serializer upgraded.
> - Compatible, but serialization schema changed: serializer upgraded but requires state
conversion, without the requirement that the old serializer needs to be present.
> - Incompatible: serializer upgraded requires state conversion, but requires the old serializer
to be present (i.e., can not be the dummy {{ClassNotFoundProxySerializer}}).

This message was sent by Atlassian JIRA

View raw message