flink-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Tzu-Li (Gordon) Tai (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (FLINK-6190) Write "Serializer Configurations" metainfo along with state
Date Sun, 07 May 2017 19:50:04 GMT

     [ https://issues.apache.org/jira/browse/FLINK-6190?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel

Tzu-Li (Gordon) Tai updated FLINK-6190:
    Fix Version/s: 1.3.0

> Write "Serializer Configurations" metainfo along with state
> -----------------------------------------------------------
>                 Key: FLINK-6190
>                 URL: https://issues.apache.org/jira/browse/FLINK-6190
>             Project: Flink
>          Issue Type: Sub-task
>          Components: State Backends, Checkpointing, Type Serialization System
>            Reporter: Tzu-Li (Gordon) Tai
>            Assignee: Tzu-Li (Gordon) Tai
>             Fix For: 1.3.0
> In order for serializers to be able to be reconfigured on restore, we need knowledge
of the previous serializer configurations, e.g. what types were registered, with which specific
/ default serializers, and especially for Kryo, the order they were registered.
> For this, we will need serializer configuration metainfo to be self-contained within
the written state.
> For the implementation, we propose to include the following changes:
> - Have a new separate {{SerializersConfig}} class that is extracted from {{ExecutionConfig}}.
This new class should contain only the serializer-related configurations (e.g., {{registeredKryoTypes}},
{{registeredPojoTypes}}, etc.). The {{SerializersConfig}} class should only be internally
used, and therefore annotated with {{Internal}}. Users should still use the {{ExecutionConfig}}
to configure serializers.
> - For serializers that previously require a {{ExecutionConfig}} in constructors, try
changing them to take a {{SerializersConfig}} instead.
> - Introduce {{SerializersConfigSerializationProxy}}, which is in charge of serializing
the current {{SerializersConfig}} when writing state to streams. This proxy defines the the
serialized format of serializer configurations, therefore should  be a {{VersionedIOReadableWritable}}
as we may change the format / information written in the future.
> - Add {{SerializersConfigSerializationProxy}} into state backends serialization proxies
(e.g. {{KeyedBackendSerializationProxy}}) so that the serializer configuration is written
into state. Need to additionally make sure backwards compatibility of previous-version backend
serialization proxies.
> For the initial version, we propose to include the following within the written serialization
config metadata (ordered):
> 1. {{registeredPojoTypes}}
> 2. {{Throwable.class}} --> {{org.apache.flink.api.java.typeutils.runtime.kryo.JavaSerializer}}
default Flink-specific registry for serializing throwables.
> 3. {{defaultKryoSerializers}}
> 4. {{defaultKryoSerializerClasses}}
> 5. Kryo registrations for all primitive types (and boxed versions). This is to allow
compatibility in case the built-in registrations for the primitive types change in Kryo in
the future.
> 6. {{registeredTypesWithKryoSerializers}}
> 7. {{registeredTypesWithKryoSerializerClasses}}

This message was sent by Atlassian JIRA

View raw message