Return-Path: X-Original-To: archive-asf-public-internal@cust-asf2.ponee.io Delivered-To: archive-asf-public-internal@cust-asf2.ponee.io Received: from cust-asf.ponee.io (cust-asf.ponee.io [163.172.22.183]) by cust-asf2.ponee.io (Postfix) with ESMTP id E5E0C200C3A for ; Fri, 3 Mar 2017 02:20:08 +0100 (CET) Received: by cust-asf.ponee.io (Postfix) id E461E160B7A; Fri, 3 Mar 2017 01:20:08 +0000 (UTC) Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by cust-asf.ponee.io (Postfix) with SMTP id 3A9E9160B6F for ; Fri, 3 Mar 2017 02:20:08 +0100 (CET) Received: (qmail 89821 invoked by uid 500); 3 Mar 2017 01:20:07 -0000 Mailing-List: contact reviews-help@impala.incubator.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Delivered-To: mailing list reviews@impala.incubator.apache.org Received: (qmail 89810 invoked by uid 99); 3 Mar 2017 01:20:07 -0000 Received: from pnap-us-west-generic-nat.apache.org (HELO spamd3-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 03 Mar 2017 01:20:07 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd3-us-west.apache.org (ASF Mail Server at spamd3-us-west.apache.org) with ESMTP id A5F9018D4A3 for ; Fri, 3 Mar 2017 01:20:06 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd3-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: 0.362 X-Spam-Level: X-Spam-Status: No, score=0.362 tagged_above=-999 required=6.31 tests=[RDNS_DYNAMIC=0.363, SPF_PASS=-0.001] autolearn=disabled Received: from mx1-lw-us.apache.org ([10.40.0.8]) by localhost (spamd3-us-west.apache.org [10.40.0.10]) (amavisd-new, port 10024) with ESMTP id nCQF9k6dOVSz for ; Fri, 3 Mar 2017 01:20:05 +0000 (UTC) Received: from ip-10-146-233-104.ec2.internal (ec2-75-101-130-251.compute-1.amazonaws.com [75.101.130.251]) by mx1-lw-us.apache.org (ASF Mail Server at mx1-lw-us.apache.org) with ESMTPS id 8B46A5FB61 for ; Fri, 3 Mar 2017 01:20:05 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by ip-10-146-233-104.ec2.internal (8.14.4/8.14.4) with ESMTP id v231K5Rn004216; Fri, 3 Mar 2017 01:20:05 GMT Message-Id: <201703030120.v231K5Rn004216@ip-10-146-233-104.ec2.internal> Date: Fri, 3 Mar 2017 01:20:05 +0000 From: "Impala Public Jenkins (Code Review)" To: Joe McDonnell , impala-cr@cloudera.com, reviews@impala.incubator.apache.org X-Gerrit-MessageType: merged Subject: =?UTF-8?Q?=5BImpala-ASF-CR=5D_IMPALA-4899=3A_Fix_parquet_table_writer_dictionary_leak=0A?= X-Gerrit-Change-Id: I06e354086ad24071d4fbf823f25f5df23933688f X-Gerrit-ChangeURL: X-Gerrit-Commit: 642b8f1b5d5493dc9e3aa55a973ef92094d4dbc9 In-Reply-To: References: MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Content-Disposition: inline User-Agent: Gerrit/2.12.7 archived-at: Fri, 03 Mar 2017 01:20:09 -0000 Impala Public Jenkins has submitted this change and it was merged. Change subject: IMPALA-4899: Fix parquet table writer dictionary leak ...................................................................... IMPALA-4899: Fix parquet table writer dictionary leak Currently, in HdfsTableSink, OutputPartitions are added to the RuntimeState object pool to be freed at the end of the query. However, for clustered inserts into a partitioned table, the OutputPartitions are only used one at a time. They can be immediately freed once done writing to that partition. In addition, the HdfsParquetTableWriter's ColumnWriters are also added to this object pool. These constitute a significant amount of memory, as they contain the dictionaries for Parquet encoding. This change makes HdfsParquetTableWriter's ColumnWriters use unique_ptrs so that they are cleaned up when the HdfsParquetTableWriter is deleted. It also uses a unique_ptr on the PartitionPair for the OutputPartition. The table writers maintain a pointer to the OutputPartition. This remains a raw pointer. This is safe, because OutputPartition has a scoped_ptr to the table writer. The table writer will never outlive the OutputPartition. Change-Id: I06e354086ad24071d4fbf823f25f5df23933688f Reviewed-on: http://gerrit.cloudera.org:8080/6181 Reviewed-by: Marcel Kornacker Tested-by: Impala Public Jenkins --- M be/src/exec/hdfs-parquet-table-writer.cc M be/src/exec/hdfs-parquet-table-writer.h M be/src/exec/hdfs-table-sink.cc M be/src/exec/hdfs-table-sink.h M be/src/exec/hdfs-table-writer.h 5 files changed, 26 insertions(+), 14 deletions(-) Approvals: Marcel Kornacker: Looks good to me, approved Impala Public Jenkins: Verified -- To view, visit http://gerrit.cloudera.org:8080/6181 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-MessageType: merged Gerrit-Change-Id: I06e354086ad24071d4fbf823f25f5df23933688f Gerrit-PatchSet: 7 Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-Owner: Joe McDonnell Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Lars Volker Gerrit-Reviewer: Marcel Kornacker Gerrit-Reviewer: Tim Armstrong