Return-Path: X-Original-To: archive-asf-public-internal@cust-asf2.ponee.io Delivered-To: archive-asf-public-internal@cust-asf2.ponee.io Received: from cust-asf.ponee.io (cust-asf.ponee.io [163.172.22.183]) by cust-asf2.ponee.io (Postfix) with ESMTP id 036A6200C41 for ; Fri, 24 Mar 2017 17:22:57 +0100 (CET) Received: by cust-asf.ponee.io (Postfix) id F32F8160B93; Fri, 24 Mar 2017 16:22:56 +0000 (UTC) Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by cust-asf.ponee.io (Postfix) with SMTP id 4CD09160B96 for ; Fri, 24 Mar 2017 17:22:56 +0100 (CET) Received: (qmail 80819 invoked by uid 500); 24 Mar 2017 16:22:55 -0000 Mailing-List: contact commits-help@flink.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@flink.apache.org Delivered-To: mailing list commits@flink.apache.org Received: (qmail 80807 invoked by uid 99); 24 Mar 2017 16:22:55 -0000 Received: from git1-us-west.apache.org (HELO git1-us-west.apache.org) (140.211.11.23) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 24 Mar 2017 16:22:55 +0000 Received: by git1-us-west.apache.org (ASF Mail Server at git1-us-west.apache.org, from userid 33) id 5ADBEDFE5C; Fri, 24 Mar 2017 16:22:55 +0000 (UTC) Content-Type: text/plain; charset="us-ascii" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit From: greg@apache.org To: commits@flink.apache.org Date: Fri, 24 Mar 2017 16:22:55 -0000 Message-Id: X-Mailer: ASF-Git Admin Mailer Subject: [1/2] flink git commit: [FLINK-5890] [gelly] GatherSumApply broken when object reuse enabled archived-at: Fri, 24 Mar 2017 16:22:57 -0000 Repository: flink Updated Branches: refs/heads/master 4b19e2720 -> 976e03c1e [FLINK-5890] [gelly] GatherSumApply broken when object reuse enabled The initial fix for this ticket is not working on larger data sets. Reduce supports returning the left input, right input, a new object, or a locally reused object. The trouble with the initial fix was that the returned local object was reusing fields from the input tuples. The problem is with ReduceDriver#run managing two values (reuse1 and reuse2) and with a third, local value returned by GatherSumApplyIteration.SumUDF. After the first grouping value.f1 == reuse1.f1. Following UDF calls may swap value.f1 and reuse2.f1, which causes reuse1.f1 == reuse2.f1. With an odd number of swaps the next grouping will reduce with reuse1 and reuse2 sharing a field and deserialization will overwrite stored values. The simple fix is to only use and return the provided inputs. This closes #3515 Project: http://git-wip-us.apache.org/repos/asf/flink/repo Commit: http://git-wip-us.apache.org/repos/asf/flink/commit/524b20f2 Tree: http://git-wip-us.apache.org/repos/asf/flink/tree/524b20f2 Diff: http://git-wip-us.apache.org/repos/asf/flink/diff/524b20f2 Branch: refs/heads/master Commit: 524b20f2db70fc4afba3a539fbf249c6d768ab4f Parents: 4b19e27 Author: Greg Hogan Authored: Fri Mar 10 16:44:27 2017 -0500 Committer: Greg Hogan Committed: Fri Mar 24 11:03:12 2017 -0400 ---------------------------------------------------------------------- .../org/apache/flink/graph/gsa/GatherSumApplyIteration.java | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) ---------------------------------------------------------------------- http://git-wip-us.apache.org/repos/asf/flink/blob/524b20f2/flink-libraries/flink-gelly/src/main/java/org/apache/flink/graph/gsa/GatherSumApplyIteration.java ---------------------------------------------------------------------- diff --git a/flink-libraries/flink-gelly/src/main/java/org/apache/flink/graph/gsa/GatherSumApplyIteration.java b/flink-libraries/flink-gelly/src/main/java/org/apache/flink/graph/gsa/GatherSumApplyIteration.java index e941b7b..5c07a73 100644 --- a/flink-libraries/flink-gelly/src/main/java/org/apache/flink/graph/gsa/GatherSumApplyIteration.java +++ b/flink-libraries/flink-gelly/src/main/java/org/apache/flink/graph/gsa/GatherSumApplyIteration.java @@ -330,7 +330,6 @@ public class GatherSumApplyIteration implements CustomUnaryOperati @Override public Tuple2 reduce(Tuple2 arg0, Tuple2 arg1) throws Exception { - K key = arg0.f0; M result = this.sumFunction.sum(arg0.f1, arg1.f1); // if the user returns value from the right argument then swap as @@ -339,9 +338,11 @@ public class GatherSumApplyIteration implements CustomUnaryOperati M tmp = arg1.f1; arg1.f1 = arg0.f1; arg0.f1 = tmp; + } else { + arg0.f1 = result; } - return new Tuple2<>(key, result); + return arg0; } @Override