Return-Path: X-Original-To: archive-asf-public-internal@cust-asf2.ponee.io Delivered-To: archive-asf-public-internal@cust-asf2.ponee.io Received: from cust-asf.ponee.io (cust-asf.ponee.io [163.172.22.183]) by cust-asf2.ponee.io (Postfix) with ESMTP id 16520200CAE for ; Wed, 21 Jun 2017 22:56:07 +0200 (CEST) Received: by cust-asf.ponee.io (Postfix) id 15431160BD5; Wed, 21 Jun 2017 20:56:07 +0000 (UTC) Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by cust-asf.ponee.io (Postfix) with SMTP id 5776A160BD0 for ; Wed, 21 Jun 2017 22:56:06 +0200 (CEST) Received: (qmail 74506 invoked by uid 500); 21 Jun 2017 20:56:05 -0000 Mailing-List: contact reviews-help@impala.incubator.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Delivered-To: mailing list reviews@impala.incubator.apache.org Received: (qmail 74495 invoked by uid 99); 21 Jun 2017 20:56:05 -0000 Received: from pnap-us-west-generic-nat.apache.org (HELO spamd4-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 21 Jun 2017 20:56:05 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd4-us-west.apache.org (ASF Mail Server at spamd4-us-west.apache.org) with ESMTP id B66A8C061E for ; Wed, 21 Jun 2017 20:56:04 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd4-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: 0.362 X-Spam-Level: X-Spam-Status: No, score=0.362 tagged_above=-999 required=6.31 tests=[RDNS_DYNAMIC=0.363, SPF_PASS=-0.001] autolearn=disabled Received: from mx1-lw-eu.apache.org ([10.40.0.8]) by localhost (spamd4-us-west.apache.org [10.40.0.11]) (amavisd-new, port 10024) with ESMTP id PxZxlkJuJQiD for ; Wed, 21 Jun 2017 20:56:03 +0000 (UTC) Received: from ip-10-146-233-104.ec2.internal (ec2-75-101-130-251.compute-1.amazonaws.com [75.101.130.251]) by mx1-lw-eu.apache.org (ASF Mail Server at mx1-lw-eu.apache.org) with ESMTPS id B73335F2F2 for ; Wed, 21 Jun 2017 20:56:02 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by ip-10-146-233-104.ec2.internal (8.14.4/8.14.4) with ESMTP id v5LKu1Tj003327; Wed, 21 Jun 2017 20:56:01 GMT Message-Id: <201706212056.v5LKu1Tj003327@ip-10-146-233-104.ec2.internal> Date: Wed, 21 Jun 2017 20:56:01 +0000 From: "Impala Public Jenkins (Code Review)" To: Tim Armstrong , impala-cr@cloudera.com, reviews@impala.incubator.apache.org X-Gerrit-MessageType: merged Subject: =?UTF-8?Q?=5BImpala-ASF-CR=5D_IMPALA-5497=3A_spilling_hash_joins_that_output_build_rows_hit_OOM=0A?= X-Gerrit-Change-Id: I075388d348499c5692d044ac1bc38dd8dd0b10c7 X-Gerrit-ChangeURL: X-Gerrit-Commit: fae36fc77d249c6e8ac372ae47c25747f670bc9f In-Reply-To: References: MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Content-Disposition: inline User-Agent: Gerrit/2.12.7 archived-at: Wed, 21 Jun 2017 20:56:07 -0000 Impala Public Jenkins has submitted this change and it was merged. Change subject: IMPALA-5497: spilling hash joins that output build rows hit OOM ...................................................................... IMPALA-5497: spilling hash joins that output build rows hit OOM The bug is that the join tried to bring the next spilled partition into memory while still holding onto memory from the current partition. The fix is to return earlier if the output batch is at capacity so that resources are flushed. Also reduce some of the redundancy in the loop that drives the spilling logic and catch some dropped statuses.. Testing: The failure was originally reproduced by my IMPALA-4703 patch. I was able to cause a query failure with the current code by reducing the memory limit for an existing query. Before it failed with up to 12MB of memory. Now it succeeds with 8MB or less. Ran exhaustive build. Change-Id: I075388d348499c5692d044ac1bc38dd8dd0b10c7 Reviewed-on: http://gerrit.cloudera.org:8080/7180 Reviewed-by: Dan Hecht Tested-by: Impala Public Jenkins --- M be/src/exec/partitioned-hash-join-builder.cc M be/src/exec/partitioned-hash-join-builder.h M be/src/exec/partitioned-hash-join-node-ir.cc M be/src/exec/partitioned-hash-join-node.cc M be/src/exec/partitioned-hash-join-node.h M testdata/workloads/functional-query/queries/QueryTest/spilling.test 6 files changed, 85 insertions(+), 85 deletions(-) Approvals: Impala Public Jenkins: Verified Dan Hecht: Looks good to me, approved -- To view, visit http://gerrit.cloudera.org:8080/7180 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-MessageType: merged Gerrit-Change-Id: I075388d348499c5692d044ac1bc38dd8dd0b10c7 Gerrit-PatchSet: 9 Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Dan Hecht Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: anujphadke