From issues-return-189114-archive-asf-public=cust-asf.ponee.io@flink.apache.org Tue Sep 18 10:06:09 2018 Return-Path: X-Original-To: archive-asf-public@cust-asf.ponee.io Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by mx-eu-01.ponee.io (Postfix) with SMTP id 92FFF180677 for ; Tue, 18 Sep 2018 10:06:08 +0200 (CEST) Received: (qmail 63310 invoked by uid 500); 18 Sep 2018 08:06:02 -0000 Mailing-List: contact issues-help@flink.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@flink.apache.org Delivered-To: mailing list issues@flink.apache.org Received: (qmail 63301 invoked by uid 99); 18 Sep 2018 08:06:02 -0000 Received: from pnap-us-west-generic-nat.apache.org (HELO spamd1-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 18 Sep 2018 08:06:02 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd1-us-west.apache.org (ASF Mail Server at spamd1-us-west.apache.org) with ESMTP id 3F87FC0E51 for ; Tue, 18 Sep 2018 08:06:02 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd1-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: -109.501 X-Spam-Level: X-Spam-Status: No, score=-109.501 tagged_above=-999 required=6.31 tests=[ENV_AND_HDR_SPF_MATCH=-0.5, KAM_ASCII_DIVIDERS=0.8, RCVD_IN_DNSWL_MED=-2.3, SPF_PASS=-0.001, USER_IN_DEF_SPF_WL=-7.5, USER_IN_WHITELIST=-100] autolearn=disabled Received: from mx1-lw-us.apache.org ([10.40.0.8]) by localhost (spamd1-us-west.apache.org [10.40.0.7]) (amavisd-new, port 10024) with ESMTP id jTnXhCNRyLGs for ; Tue, 18 Sep 2018 08:06:01 +0000 (UTC) Received: from mailrelay1-us-west.apache.org (mailrelay1-us-west.apache.org [209.188.14.139]) by mx1-lw-us.apache.org (ASF Mail Server at mx1-lw-us.apache.org) with ESMTP id 69E025F10C for ; Tue, 18 Sep 2018 08:06:01 +0000 (UTC) Received: from jira-lw-us.apache.org (unknown [207.244.88.139]) by mailrelay1-us-west.apache.org (ASF Mail Server at mailrelay1-us-west.apache.org) with ESMTP id 09865E02BE for ; Tue, 18 Sep 2018 08:06:01 +0000 (UTC) Received: from jira-lw-us.apache.org (localhost [127.0.0.1]) by jira-lw-us.apache.org (ASF Mail Server at jira-lw-us.apache.org) with ESMTP id CD4A923FA0 for ; Tue, 18 Sep 2018 08:06:00 +0000 (UTC) Date: Tue, 18 Sep 2018 08:06:00 +0000 (UTC) From: "ASF GitHub Bot (JIRA)" To: issues@flink.apache.org Message-ID: In-Reply-To: References: Subject: [jira] [Commented] (FLINK-9377) Remove writing serializers as part of the checkpoint meta information MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394 [ https://issues.apache.org/jira/browse/FLINK-9377?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16618644#comment-16618644 ] ASF GitHub Bot commented on FLINK-9377: --------------------------------------- tzulitai commented on issue #6711: [FLINK-9377] [core, state backends] Remove serializers from checkpoints URL: https://github.com/apache/flink/pull/6711#issuecomment-422296230 cc @StephanEwen, would be best if you can review this one since you were part of most of the initial design discussions. ---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: users@infra.apache.org > Remove writing serializers as part of the checkpoint meta information > --------------------------------------------------------------------- > > Key: FLINK-9377 > URL: https://issues.apache.org/jira/browse/FLINK-9377 > Project: Flink > Issue Type: Sub-task > Components: State Backends, Checkpointing > Reporter: Tzu-Li (Gordon) Tai > Assignee: Tzu-Li (Gordon) Tai > Priority: Critical > Labels: pull-request-available > Fix For: 1.7.0 > > > When writing meta information of a state in savepoints, we currently write both the state serializer as well as the state serializer's configuration snapshot. > Writing both is actually redundant, as most of the time they have identical information. > Moreover, the fact that we use Java serialization to write the serializer and rely on it to be re-readable on the restore run, already poses problems for serializers such as the {{AvroSerializer}} (see discussion in FLINK-9202) to perform even a compatible upgrade. > The proposal here is to leave only the config snapshot as meta information, and use that as the single source of truth of information about the schema of serialized state. > The config snapshot should be treated as a factory (or provided to a factory) to re-create serializers capable of reading old, serialized state. -- This message was sent by Atlassian JIRA (v7.6.3#76005)