From issues-return-151172-archive-asf-public=cust-asf.ponee.io@flink.apache.org Tue Feb 6 11:48:07 2018 Return-Path: X-Original-To: archive-asf-public@eu.ponee.io Delivered-To: archive-asf-public@eu.ponee.io Received: from cust-asf.ponee.io (cust-asf.ponee.io [163.172.22.183]) by mx-eu-01.ponee.io (Postfix) with ESMTP id 90420180657 for ; Tue, 6 Feb 2018 11:48:07 +0100 (CET) Received: by cust-asf.ponee.io (Postfix) id 80155160C45; Tue, 6 Feb 2018 10:48:07 +0000 (UTC) Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by cust-asf.ponee.io (Postfix) with SMTP id C79DA160C3A for ; Tue, 6 Feb 2018 11:48:06 +0100 (CET) Received: (qmail 4805 invoked by uid 500); 6 Feb 2018 10:48:06 -0000 Mailing-List: contact issues-help@flink.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@flink.apache.org Delivered-To: mailing list issues@flink.apache.org Received: (qmail 4796 invoked by uid 99); 6 Feb 2018 10:48:05 -0000 Received: from pnap-us-west-generic-nat.apache.org (HELO spamd3-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 06 Feb 2018 10:48:05 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd3-us-west.apache.org (ASF Mail Server at spamd3-us-west.apache.org) with ESMTP id 9255018032C for ; Tue, 6 Feb 2018 10:48:05 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd3-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: -110.311 X-Spam-Level: X-Spam-Status: No, score=-110.311 tagged_above=-999 required=6.31 tests=[ENV_AND_HDR_SPF_MATCH=-0.5, RCVD_IN_DNSWL_MED=-2.3, SPF_PASS=-0.001, T_RP_MATCHES_RCVD=-0.01, USER_IN_DEF_SPF_WL=-7.5, USER_IN_WHITELIST=-100] autolearn=disabled Received: from mx1-lw-us.apache.org ([10.40.0.8]) by localhost (spamd3-us-west.apache.org [10.40.0.10]) (amavisd-new, port 10024) with ESMTP id 3TTd-PMZgD09 for ; Tue, 6 Feb 2018 10:48:01 +0000 (UTC) Received: from mailrelay1-us-west.apache.org (mailrelay1-us-west.apache.org [209.188.14.139]) by mx1-lw-us.apache.org (ASF Mail Server at mx1-lw-us.apache.org) with ESMTP id C2F925F11F for ; Tue, 6 Feb 2018 10:48:00 +0000 (UTC) Received: from jira-lw-us.apache.org (unknown [207.244.88.139]) by mailrelay1-us-west.apache.org (ASF Mail Server at mailrelay1-us-west.apache.org) with ESMTP id 52628E00CA for ; Tue, 6 Feb 2018 10:48:00 +0000 (UTC) Received: from jira-lw-us.apache.org (localhost [127.0.0.1]) by jira-lw-us.apache.org (ASF Mail Server at jira-lw-us.apache.org) with ESMTP id 1269521E86 for ; Tue, 6 Feb 2018 10:48:00 +0000 (UTC) Date: Tue, 6 Feb 2018 10:48:00 +0000 (UTC) From: "ASF GitHub Bot (JIRA)" To: issues@flink.apache.org Message-ID: In-Reply-To: References: Subject: [jira] [Commented] (FLINK-8516) FlinkKinesisConsumer does not balance shards over subtasks MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394 [ https://issues.apache.org/jira/browse/FLINK-8516?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16353705#comment-16353705 ] ASF GitHub Bot commented on FLINK-8516: --------------------------------------- Github user tzulitai commented on the issue: https://github.com/apache/flink/pull/5393 @tweise regarding what the proposed migration test is going to assert: > The assigner does not influence how state is saved and restored. Even when the assigner returns invalid index, the modulus will ensure that the shard gets assigned. This is exactly what I think the test is verifying, without any knowledge of what internally is happening. Using a different custom assigner across restores should not result in any state lose / change. > FlinkKinesisConsumer does not balance shards over subtasks > ---------------------------------------------------------- > > Key: FLINK-8516 > URL: https://issues.apache.org/jira/browse/FLINK-8516 > Project: Flink > Issue Type: Bug > Components: Kinesis Connector > Affects Versions: 1.4.0, 1.3.2, 1.5.0 > Reporter: Thomas Weise > Assignee: Thomas Weise > Priority: Major > > The hash code of the shard is used to distribute discovered shards over subtasks round robin. This works as long as shard identifiers are sequential. After shards are rebalanced in Kinesis, that may no longer be the case and the distribution become skewed. -- This message was sent by Atlassian JIRA (v7.6.3#76005)