flink-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Tzu-Li (Gordon) Tai (JIRA)" <j...@apache.org>
Subject [jira] [Created] (FLINK-6190) Write "Serializer Configurations" metainfo along with state
Date Mon, 27 Mar 2017 05:39:41 GMT
Tzu-Li (Gordon) Tai created FLINK-6190:

             Summary: Write "Serializer Configurations" metainfo along with state
                 Key: FLINK-6190
                 URL: https://issues.apache.org/jira/browse/FLINK-6190
             Project: Flink
          Issue Type: Sub-task
          Components: State Backends, Checkpointing, Type Serialization System
            Reporter: Tzu-Li (Gordon) Tai
            Assignee: Tzu-Li (Gordon) Tai

In order for serializers to be able to be reconfigured on restore, we need knowledge of the
previous serializer configurations, e.g. what types were registered, with which specific /
default serializers, and especially for Kryo, the order they were registered.

For this, we will need serializer configuration metainfo to be self-contained within the written

For the implementation, we propose the following:

- Have a new separate `SerializersConfig` class that is extracted from `ExecutionConfig`.
This new class should contain only the serializer-related configurations (e.g., `registeredKryoTypes`,
`registeredPojoTypes`, etc.).

- Introduce `SerializersConfigSerializationProxy`, which is in charge of serializing the current
`SerializersConfig` when writing state to streams. This proxy defines the the serialized format
of serializer configurations, therefore should  be a `VersionedIOReadableWritable` as we may
change the format / information written in the future.

- Add `SerializersConfigSerializationProxy` into state backends serialization proxies (e.g.
`KeyedBackendSerializationProxy`) so that the serializer configuration is written into state.
Need to additionally make sure backwards compatibility of previous-version backend serialization

This message was sent by Atlassian JIRA

View raw message