flink-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "ASF GitHub Bot (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (FLINK-6653) Avoid directly serializing AWS's Shard class in Kinesis consumer's checkpoints
Date Fri, 26 May 2017 07:49:04 GMT

    [ https://issues.apache.org/jira/browse/FLINK-6653?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16025943#comment-16025943
] 

ASF GitHub Bot commented on FLINK-6653:
---------------------------------------

Github user tzulitai commented on the issue:

    https://github.com/apache/flink/pull/3994
  
    @tony810430 thanks for the PR! I'll like to merge this for 1.3, to avoid another turn
of state migration. Will merge with some minor cleanups.


> Avoid directly serializing AWS's Shard class in Kinesis consumer's checkpoints
> ------------------------------------------------------------------------------
>
>                 Key: FLINK-6653
>                 URL: https://issues.apache.org/jira/browse/FLINK-6653
>             Project: Flink
>          Issue Type: Improvement
>          Components: Kinesis Connector
>            Reporter: Tzu-Li (Gordon) Tai
>            Assignee: Wei-Che Wei
>
> Currently, the Kinesis consumer's checkpoints directly serialize AWS's {{Shard}} instances
in checkpoints. This makes bumping AWS library versions hard, since any change to the {{Shard}}
class by AWS will break checkpoint compatibility.
> We should either have custom serialization for {{KinesisStreamShard}}, or disintegrate
the information in {{Shard}}. Ideally, it would be best to make {{KinesisStreamShard}} and
{{SequenceNumber}} to be non-serializable, hence avoiding Java serialization in the checkpoints.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

Mime
View raw message