Return-Path: X-Original-To: archive-asf-public-internal@cust-asf2.ponee.io Delivered-To: archive-asf-public-internal@cust-asf2.ponee.io Received: from cust-asf.ponee.io (cust-asf.ponee.io [163.172.22.183]) by cust-asf2.ponee.io (Postfix) with ESMTP id 8065D200B58 for ; Tue, 12 Jul 2016 21:16:18 +0200 (CEST) Received: by cust-asf.ponee.io (Postfix) id 7F0BE160A53; Tue, 12 Jul 2016 19:16:18 +0000 (UTC) Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by cust-asf.ponee.io (Postfix) with SMTP id CFD8F160A56 for ; Tue, 12 Jul 2016 21:16:17 +0200 (CEST) Received: (qmail 55294 invoked by uid 500); 12 Jul 2016 19:16:17 -0000 Mailing-List: contact commits-help@beam.incubator.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@beam.incubator.apache.org Delivered-To: mailing list commits@beam.incubator.apache.org Received: (qmail 55274 invoked by uid 99); 12 Jul 2016 19:16:17 -0000 Received: from pnap-us-west-generic-nat.apache.org (HELO spamd1-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 12 Jul 2016 19:16:17 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd1-us-west.apache.org (ASF Mail Server at spamd1-us-west.apache.org) with ESMTP id 7FE90C0C3B for ; Tue, 12 Jul 2016 19:16:16 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd1-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: -4.507 X-Spam-Level: X-Spam-Status: No, score=-4.507 tagged_above=-999 required=6.31 tests=[KAM_ASCII_DIVIDERS=0.8, KAM_LAZY_DOMAIN_SECURITY=1, RCVD_IN_DNSWL_HI=-5, RCVD_IN_MSPIKE_H3=-0.01, RCVD_IN_MSPIKE_WL=-0.01, RP_MATCHES_RCVD=-1.287] autolearn=disabled Received: from mx2-lw-eu.apache.org ([10.40.0.8]) by localhost (spamd1-us-west.apache.org [10.40.0.7]) (amavisd-new, port 10024) with ESMTP id lUniYZuAhSLm for ; Tue, 12 Jul 2016 19:16:14 +0000 (UTC) Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by mx2-lw-eu.apache.org (ASF Mail Server at mx2-lw-eu.apache.org) with SMTP id D86795F474 for ; Tue, 12 Jul 2016 19:16:13 +0000 (UTC) Received: (qmail 55164 invoked by uid 99); 12 Jul 2016 19:16:13 -0000 Received: from git1-us-west.apache.org (HELO git1-us-west.apache.org) (140.211.11.23) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 12 Jul 2016 19:16:13 +0000 Received: by git1-us-west.apache.org (ASF Mail Server at git1-us-west.apache.org, from userid 33) id 0FD2BE1021; Tue, 12 Jul 2016 19:16:13 +0000 (UTC) Content-Type: text/plain; charset="us-ascii" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit From: robertwb@apache.org To: commits@beam.incubator.apache.org Date: Tue, 12 Jul 2016 19:16:14 -0000 Message-Id: <986ed4f876c049f2bcc9ced93e9c0589@git.apache.org> In-Reply-To: References: X-Mailer: ASF-Git Admin Mailer Subject: [2/2] incubator-beam git commit: Fixes bug due to accessing cached pvalues multiple times. archived-at: Tue, 12 Jul 2016 19:16:18 -0000 Fixes bug due to accessing cached pvalues multiple times. This lookup is not an idempotent operation, as it decrements the refcount and may even delete the pvalue from the cache. Instead, we do the lookup once, storing the result in a map that is referenced elsewhere. Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/246fda51 Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/246fda51 Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/246fda51 Branch: refs/heads/python-sdk Commit: 246fda517fd7d6abdbbd47882657e66c34a4ac51 Parents: 77f90ff Author: Robert Bradshaw Authored: Tue Jul 12 10:43:29 2016 -0700 Committer: Robert Bradshaw Committed: Tue Jul 12 12:15:25 2016 -0700 ---------------------------------------------------------------------- sdks/python/apache_beam/runners/dataflow_runner.py | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-) ---------------------------------------------------------------------- http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/246fda51/sdks/python/apache_beam/runners/dataflow_runner.py ---------------------------------------------------------------------- diff --git a/sdks/python/apache_beam/runners/dataflow_runner.py b/sdks/python/apache_beam/runners/dataflow_runner.py index 24edb05..5a3f6a5 100644 --- a/sdks/python/apache_beam/runners/dataflow_runner.py +++ b/sdks/python/apache_beam/runners/dataflow_runner.py @@ -361,7 +361,11 @@ class DataflowPipelineRunner(PipelineRunner): # Attach side inputs. si_dict = {} - lookup_label = lambda side_pval: self._cache.get_pvalue(side_pval).step_name + # We must call self._cache.get_pvalue exactly once due to refcounting. + si_labels = {} + for side_pval in transform_node.side_inputs: + si_labels[side_pval] = self._cache.get_pvalue(side_pval).step_name + lookup_label = lambda side_pval: si_labels[side_pval] for side_pval in transform_node.side_inputs: assert isinstance(side_pval, PCollectionView) si_label = lookup_label(side_pval)