flink-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Tzu-Li (Gordon) Tai (JIRA)" <j...@apache.org>
Subject [jira] [Created] (FLINK-6349) Enforce per-subtask record ordering on resharding for FlinkKinesisConsumer
Date Fri, 21 Apr 2017 00:08:04 GMT
Tzu-Li (Gordon) Tai created FLINK-6349:

             Summary: Enforce per-subtask record ordering on resharding for FlinkKinesisConsumer
                 Key: FLINK-6349
                 URL: https://issues.apache.org/jira/browse/FLINK-6349
             Project: Flink
          Issue Type: Improvement
          Components: Kinesis Connector
            Reporter: Tzu-Li (Gordon) Tai

As described in FLINK-6316, currently the Kinesis consumer does not provide any ordering guarantees
when resharding occurs.

While this cannot be enforced globally (i.e. if a merged / split shard's child shard ends
up in a different subtask, we cannot do any coordination for ordering guarantee), we can definitely
enforce this locally for each subtask. Simply put, we can still locally enforce ordering by
making sure that discovered child shards are consumed only after any of its parent shards
that were on the same subtask are fully consumed.

To do this, we would also need to add "parent shard" information to {{KinesisStreamShard}}
(Flink's representation of Kinesis shards).

This would be directly beneficial for per-shard watermarks (FLINK-5697) to retain per-shard
time characteristics after a reshard, and therefore can be seen as a prerequisite.

This message was sent by Atlassian JIRA

View raw message