flink-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Tzu-Li (Gordon) Tai (JIRA)" <j...@apache.org>
Subject [jira] [Created] (FLINK-6653) Avoid directly serializing AWS's Shard class in Kinesis consumer's checkpoints
Date Mon, 22 May 2017 05:26:04 GMT
Tzu-Li (Gordon) Tai created FLINK-6653:
------------------------------------------

             Summary: Avoid directly serializing AWS's Shard class in Kinesis consumer's checkpoints
                 Key: FLINK-6653
                 URL: https://issues.apache.org/jira/browse/FLINK-6653
             Project: Flink
          Issue Type: Improvement
          Components: Kinesis Connector
            Reporter: Tzu-Li (Gordon) Tai


Currently, the Kinesis consumer's checkpoints directly serialize AWS's `Shard` instances in
checkpoints. This makes bumping AWS library versions hard, since any change to the `Shard`
class by AWS will break checkpoint compatibility.

We should either have custom serialization for `KinesisStreamShard`, or disintegrate the information
in `Shard`. Ideally, it would be best to make `KinesisStreamShard` and `SequenceNumber` to
be non-serializable, hence avoiding Java serialization in the checkpoints.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

Mime
View raw message