flink-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Tzu-Li (Gordon) Tai (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (FLINK-8516) FlinkKinesisConsumer does not balance shards over subtasks
Date Fri, 26 Jan 2018 19:09:04 GMT

    [ https://issues.apache.org/jira/browse/FLINK-8516?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16341450#comment-16341450
] 

Tzu-Li (Gordon) Tai commented on FLINK-8516:
--------------------------------------------

Posting the mailing list discussion thread of the issue here, for easier reference:
http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/Kinesis-consumer-shard-skew-FLINK-8516-td20843.html

> FlinkKinesisConsumer does not balance shards over subtasks
> ----------------------------------------------------------
>
>                 Key: FLINK-8516
>                 URL: https://issues.apache.org/jira/browse/FLINK-8516
>             Project: Flink
>          Issue Type: Bug
>          Components: Kinesis Connector
>    Affects Versions: 1.4.0, 1.3.2, 1.5.0
>            Reporter: Thomas Weise
>            Assignee: Thomas Weise
>            Priority: Major
>
> The hash code of the shard is used to distribute discovered shards over subtasks round
robin. This works as long as shard identifiers are sequential. After shards are rebalanced
in Kinesis, that may no longer be the case and the distribution become skewed.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Mime
View raw message