flink-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "ASF GitHub Bot (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (FLINK-8516) FlinkKinesisConsumer does not balance shards over subtasks
Date Tue, 06 Feb 2018 10:48:00 GMT

    [ https://issues.apache.org/jira/browse/FLINK-8516?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16353705#comment-16353705
] 

ASF GitHub Bot commented on FLINK-8516:
---------------------------------------

Github user tzulitai commented on the issue:

    https://github.com/apache/flink/pull/5393
  
    @tweise regarding what the proposed migration test is going to assert:
    
    > The assigner does not influence how state is saved and restored. Even when the assigner
returns invalid index, the modulus will ensure that the shard gets assigned.
    
    This is exactly what I think the test is verifying, without any knowledge of what internally
is happening. Using a different custom assigner across restores should not result in any state
lose / change.


> FlinkKinesisConsumer does not balance shards over subtasks
> ----------------------------------------------------------
>
>                 Key: FLINK-8516
>                 URL: https://issues.apache.org/jira/browse/FLINK-8516
>             Project: Flink
>          Issue Type: Bug
>          Components: Kinesis Connector
>    Affects Versions: 1.4.0, 1.3.2, 1.5.0
>            Reporter: Thomas Weise
>            Assignee: Thomas Weise
>            Priority: Major
>
> The hash code of the shard is used to distribute discovered shards over subtasks round
robin. This works as long as shard identifiers are sequential. After shards are rebalanced
in Kinesis, that may no longer be the case and the distribution become skewed.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Mime
View raw message