Return-Path: X-Original-To: apmail-flink-dev-archive@www.apache.org Delivered-To: apmail-flink-dev-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id E37961845A for ; Thu, 4 Feb 2016 17:28:54 +0000 (UTC) Received: (qmail 86519 invoked by uid 500); 4 Feb 2016 17:28:40 -0000 Delivered-To: apmail-flink-dev-archive@flink.apache.org Received: (qmail 86440 invoked by uid 500); 4 Feb 2016 17:28:40 -0000 Mailing-List: contact dev-help@flink.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@flink.apache.org Delivered-To: mailing list dev@flink.apache.org Received: (qmail 86325 invoked by uid 99); 4 Feb 2016 17:28:40 -0000 Received: from arcas.apache.org (HELO arcas) (140.211.11.28) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 04 Feb 2016 17:28:40 +0000 Received: from arcas.apache.org (localhost [127.0.0.1]) by arcas (Postfix) with ESMTP id EFF112C1F5D for ; Thu, 4 Feb 2016 17:28:39 +0000 (UTC) Date: Thu, 4 Feb 2016 17:28:39 +0000 (UTC) From: "Aljoscha Krettek (JIRA)" To: dev@flink.apache.org Message-ID: In-Reply-To: References: Subject: [jira] [Created] (FLINK-3336) Add Semi-Rebalance Data Shipping for DataStream MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394 Aljoscha Krettek created FLINK-3336: --------------------------------------- Summary: Add Semi-Rebalance Data Shipping for DataStream Key: FLINK-3336 URL: https://issues.apache.org/jira/browse/FLINK-3336 Project: Flink Issue Type: Improvement Components: Streaming Reporter: Aljoscha Krettek Assignee: Aljoscha Krettek Fix For: 1.0.0 This feature has recently been requested on the ML: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Distribution-of-sinks-among-the-nodes-td4640.html The new data shipping pattern would allow to rebalance data only to a subset of downstream operations. The subset of downstream operations to which the upstream operation would send elements depends on the degree of parallelism of both the upstream and downstream operation. For example, if the upstream operation has parallelism 2 and the downstream operation has parallelism 4, then one upstream operation would distribute elements to two downstream operations while the other upstream operation would distribute to the other two downstream operations. If, on the other hand, the downstream operation had parallelism 2 while the upstream operation has parallelism 4 then two upstream operations would distribute to one downstream operation while the other two upstream operations would distribute to the other downstream operations. In cases where the different parallelisms are not multiples of each other one or several downstream operations would have a differing number of inputs from upstream operations. -- This message was sent by Atlassian JIRA (v6.3.4#6332)