flink-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Tzu-Li (Gordon) Tai (JIRA)" <j...@apache.org>
Subject [jira] [Created] (FLINK-8542) Do not indefinitely store closed shard's state in the FlinkKinesisConsumer
Date Thu, 01 Feb 2018 10:59:00 GMT
Tzu-Li (Gordon) Tai created FLINK-8542:
------------------------------------------

             Summary: Do not indefinitely store closed shard's state in the FlinkKinesisConsumer
                 Key: FLINK-8542
                 URL: https://issues.apache.org/jira/browse/FLINK-8542
             Project: Flink
          Issue Type: Improvement
          Components: Kinesis Connector
            Reporter: Tzu-Li (Gordon) Tai


See original discussion here: [https://github.com/apache/flink/pull/5337|https://github.com/apache/flink/pull/5337#issuecomment-362227711]

Currently, the Kinesis consumer keeps a list of {{(StreamShardMetadata, SequenceNumber)}}
as its state. That list also contains all shards that have been closed already, and is kept
in the state indefinitely so that on restore, we know that a closed shard is already fully
consumed,

The downside of this, is that the state size of the Kinesis consumer can basically grow without
bounds, as the consumed Kinesis streams are resharded and more and more closed shards are
present.

Some possible solutions have been discussed in the linked PR comments.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Mime
View raw message