Mailing-List: contact issues-help@flink.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@flink.apache.org
Date: Mon, 27 Mar 2017 05:45:42 +0000 (UTC)
From: "Tzu-Li (Gordon) Tai (JIRA)" <jira@apache.org>
To: issues@flink.apache.org
Message-ID: <JIRA.13059284.1490593175000.128038.1490593542024@Atlassian.JIRA>
In-Reply-To: <JIRA.13059284.1490593175000@Atlassian.JIRA>
References: <JIRA.13059284.1490593175000@Atlassian.JIRA> <JIRA.13059284.1490593175913@jira-lw-us.apache.org>
Subject: [jira] [Updated] (FLINK-6190) Write "Serializer Configurations"
 metainfo along with state
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
archived-at: Mon, 27 Mar 2017 05:45:48 -0000


     [ https://issues.apache.org/jira/browse/FLINK-6190?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Tzu-Li (Gordon) Tai updated FLINK-6190:
---------------------------------------
    Description: 
In order for serializers to be able to be reconfigured on restore, we need knowledge of the previous serializer configurations, e.g. what types were registered, with which specific / default serializers, and especially for Kryo, the order they were registered.

For this, we will need serializer configuration metainfo to be self-contained within the written state.

For the implementation, we propose to include the following changes:

- Have a new separate {{SerializersConfig}} class that is extracted from {{ExecutionConfig}}. This new class should contain only the serializer-related configurations (e.g., {{registeredKryoTypes}}, {{registeredPojoTypes}}, etc.). The {{SerializersConfig}} class should only be internally used, and therefore annotated with {{Internal}}. Users should still use the {{ExecutionConfig}} to configure serializers.

- For serializers that previously require a {{ExecutionConfig}} in constructors, try changing them to take a {{SerializersConfig}} instead.

- Introduce {{SerializersConfigSerializationProxy}}, which is in charge of serializing the current {{SerializersConfig}} when writing state to streams. This proxy defines the the serialized format of serializer configurations, therefore should  be a {{VersionedIOReadableWritable}} as we may change the format / information written in the future.

- Add {{SerializersConfigSerializationProxy}} into state backends serialization proxies (e.g. {{KeyedBackendSerializationProxy}}) so that the serializer configuration is written into state. Need to additionally make sure backwards compatibility of previous-version backend serialization proxies.

  was:
In order for serializers to be able to be reconfigured on restore, we need knowledge of the previous serializer configurations, e.g. what types were registered, with which specific / default serializers, and especially for Kryo, the order they were registered.

For this, we will need serializer configuration metainfo to be self-contained within the written state.

For the implementation, we propose the following:

- Have a new separate {{SerializersConfig}} class that is extracted from {{ExecutionConfig}}. This new class should contain only the serializer-related configurations (e.g., {{registeredKryoTypes}}, {{registeredPojoTypes}}, etc.).

- Introduce {{SerializersConfigSerializationProxy}}, which is in charge of serializing the current {{SerializersConfig}} when writing state to streams. This proxy defines the the serialized format of serializer configurations, therefore should  be a {{VersionedIOReadableWritable}} as we may change the format / information written in the future.

- Add {{SerializersConfigSerializationProxy}} into state backends serialization proxies (e.g. {{KeyedBackendSerializationProxy}}) so that the serializer configuration is written into state. Need to additionally make sure backwards compatibility of previous-version backend serialization proxies.


> Write "Serializer Configurations" metainfo along with state
> -----------------------------------------------------------
>
>                 Key: FLINK-6190
>                 URL: https://issues.apache.org/jira/browse/FLINK-6190
>             Project: Flink
>          Issue Type: Sub-task
>          Components: State Backends, Checkpointing, Type Serialization System
>            Reporter: Tzu-Li (Gordon) Tai
>            Assignee: Tzu-Li (Gordon) Tai
>
> In order for serializers to be able to be reconfigured on restore, we need knowledge of the previous serializer configurations, e.g. what types were registered, with which specific / default serializers, and especially for Kryo, the order they were registered.
> For this, we will need serializer configuration metainfo to be self-contained within the written state.
> For the implementation, we propose to include the following changes:
> - Have a new separate {{SerializersConfig}} class that is extracted from {{ExecutionConfig}}. This new class should contain only the serializer-related configurations (e.g., {{registeredKryoTypes}}, {{registeredPojoTypes}}, etc.). The {{SerializersConfig}} class should only be internally used, and therefore annotated with {{Internal}}. Users should still use the {{ExecutionConfig}} to configure serializers.
> - For serializers that previously require a {{ExecutionConfig}} in constructors, try changing them to take a {{SerializersConfig}} instead.
> - Introduce {{SerializersConfigSerializationProxy}}, which is in charge of serializing the current {{SerializersConfig}} when writing state to streams. This proxy defines the the serialized format of serializer configurations, therefore should  be a {{VersionedIOReadableWritable}} as we may change the format / information written in the future.
> - Add {{SerializersConfigSerializationProxy}} into state backends serialization proxies (e.g. {{KeyedBackendSerializationProxy}}) so that the serializer configuration is written into state. Need to additionally make sure backwards compatibility of previous-version backend serialization proxies.


--
This message was sent by Atlassian JIRA
(v6.3.15#6346)