Return-Path: X-Original-To: archive-asf-public-internal@cust-asf2.ponee.io Delivered-To: archive-asf-public-internal@cust-asf2.ponee.io Received: from cust-asf.ponee.io (cust-asf.ponee.io [163.172.22.183]) by cust-asf2.ponee.io (Postfix) with ESMTP id C6DEF200B68 for ; Fri, 19 Aug 2016 09:02:46 +0200 (CEST) Received: by cust-asf.ponee.io (Postfix) id C56AE160AAC; Fri, 19 Aug 2016 07:02:46 +0000 (UTC) Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by cust-asf.ponee.io (Postfix) with SMTP id 17F82160A8E for ; Fri, 19 Aug 2016 09:02:45 +0200 (CEST) Received: (qmail 95532 invoked by uid 500); 19 Aug 2016 07:02:45 -0000 Mailing-List: contact dev-help@impala.incubator.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@impala.incubator.apache.org Delivered-To: mailing list dev@impala.incubator.apache.org Received: (qmail 95517 invoked by uid 99); 19 Aug 2016 07:02:44 -0000 Received: from pnap-us-west-generic-nat.apache.org (HELO spamd1-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 19 Aug 2016 07:02:44 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd1-us-west.apache.org (ASF Mail Server at spamd1-us-west.apache.org) with ESMTP id 97839C12D2 for ; Fri, 19 Aug 2016 07:02:44 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd1-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: 0.362 X-Spam-Level: X-Spam-Status: No, score=0.362 tagged_above=-999 required=6.31 tests=[RDNS_DYNAMIC=0.363, SPF_PASS=-0.001] autolearn=disabled Received: from mx1-lw-us.apache.org ([10.40.0.8]) by localhost (spamd1-us-west.apache.org [10.40.0.7]) (amavisd-new, port 10024) with ESMTP id 0-BvMLHMfLyN for ; Fri, 19 Aug 2016 07:02:41 +0000 (UTC) Received: from ip-10-146-233-104.ec2.internal (ec2-75-101-130-251.compute-1.amazonaws.com [75.101.130.251]) by mx1-lw-us.apache.org (ASF Mail Server at mx1-lw-us.apache.org) with ESMTPS id 76D8B5F571 for ; Fri, 19 Aug 2016 07:02:40 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by ip-10-146-233-104.ec2.internal (8.14.4/8.14.4) with ESMTP id u7J72cqc014596; Fri, 19 Aug 2016 07:02:38 GMT Message-Id: <201608190702.u7J72cqc014596@ip-10-146-233-104.ec2.internal> Date: Fri, 19 Aug 2016 07:02:38 +0000 From: "Michael Ho (Code Review)" To: impala-cr@cloudera.com, dev@impala.incubator.apache.org CC: Alex Behm , Dan Hecht Reply-To: kwho@cloudera.com X-Gerrit-MessageType: newpatchset Subject: =?UTF-8?Q?=5BImpala-ASF-CR=5D_IMPALA-3962=3A_Clean_up_scratch_tuple_batch_on_scan_failures=0A?= X-Gerrit-Change-Id: If1e27a1517d09ccaabdae1492b7e1fbf661ae3e5 X-Gerrit-ChangeURL: X-Gerrit-Commit: 4436c1ab995851d1e14e28c71eb3d102093b5e17 In-Reply-To: References: MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Content-Disposition: inline User-Agent: Gerrit/2.12.2 archived-at: Fri, 19 Aug 2016 07:02:47 -0000 Michael Ho has uploaded a new patch set (#4). Change subject: IMPALA-3962: Clean up scratch tuple batch on scan failures ...................................................................... IMPALA-3962: Clean up scratch tuple batch on scan failures The parquet scanner doesn't clean up 'scratch_batch_' properly which causes it to process a partially filled scratch_batch_ if any of the column reader fails. This change cleans up the scratch batch if any of the parquet column readers fails. The clean up involves freeing the mem_pool of scratch_batch_ and setting number of tuples in scratch_batch_ to 0. This change also extends debug action to emulate the behavior of exceeding the query's memory limit. Change-Id: If1e27a1517d09ccaabdae1492b7e1fbf661ae3e5 --- M be/src/exec/exec-node.cc M be/src/exec/hdfs-parquet-scanner.cc M be/src/exec/hdfs-scan-node.cc M be/src/exec/hdfs-scan-node.h M be/src/exec/hdfs-scanner.cc M be/src/exec/hdfs-scanner.h M be/src/exec/parquet-column-readers.cc M be/src/exec/parquet-column-readers.h M common/thrift/PlanNodes.thrift M tests/failure/test_failpoints.py 10 files changed, 77 insertions(+), 22 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/91/3991/4 -- To view, visit http://gerrit.cloudera.org:8080/3991 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-MessageType: newpatchset Gerrit-Change-Id: If1e27a1517d09ccaabdae1492b7e1fbf661ae3e5 Gerrit-PatchSet: 4 Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-Owner: Michael Ho Gerrit-Reviewer: Alex Behm Gerrit-Reviewer: Dan Hecht Gerrit-Reviewer: Michael Ho