Return-Path: X-Original-To: apmail-hive-dev-archive@www.apache.org Delivered-To: apmail-hive-dev-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 7D9C09EAF for ; Mon, 4 Mar 2013 01:12:12 +0000 (UTC) Received: (qmail 50452 invoked by uid 500); 4 Mar 2013 01:12:12 -0000 Delivered-To: apmail-hive-dev-archive@hive.apache.org Received: (qmail 50387 invoked by uid 500); 4 Mar 2013 01:12:11 -0000 Mailing-List: contact dev-help@hive.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@hive.apache.org Delivered-To: mailing list dev@hive.apache.org Received: (qmail 50377 invoked by uid 500); 4 Mar 2013 01:12:11 -0000 Delivered-To: apmail-hadoop-hive-dev@hadoop.apache.org Received: (qmail 50373 invoked by uid 99); 4 Mar 2013 01:12:11 -0000 Received: from arcas.apache.org (HELO arcas.apache.org) (140.211.11.28) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 04 Mar 2013 01:12:11 +0000 Date: Mon, 4 Mar 2013 01:12:11 +0000 (UTC) From: "Phabricator (JIRA)" To: hive-dev@hadoop.apache.org Message-ID: In-Reply-To: References: Subject: [jira] [Updated] (HIVE-4068) Size of aggregation buffer which uses non-primitive type is not estimated correctly MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394 [ https://issues.apache.org/jira/browse/HIVE-4068?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Phabricator updated HIVE-4068: ------------------------------ Attachment: HIVE-4068.D8859.4.patch navis updated the revision "HIVE-4068 [jira] Size of aggregation buffer which uses non-primitive type is not estimated correctly". Addressed comment Reviewers: JIRA REVISION DETAIL https://reviews.facebook.net/D8859 CHANGE SINCE LAST DIFF https://reviews.facebook.net/D8859?vs=28593&id=29133#toc AFFECTED FILES builtins/src/org/apache/hive/builtins/UDAFUnionMap.java pdk/scripts/build-plugin.xml ql/src/java/org/apache/hadoop/hive/ql/exec/GroupByOperator.java ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDAFAverage.java ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDAFBridge.java ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDAFCollectSet.java ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDAFComputeStats.java ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDAFContextNGrams.java ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDAFCorrelation.java ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDAFCount.java ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDAFCovariance.java ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDAFEWAHBitmap.java ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDAFEvaluator.java ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDAFHistogramNumeric.java ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDAFMax.java ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDAFMin.java ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDAFPercentileApprox.java ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDAFSum.java ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDAFVariance.java ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDAFnGrams.java ql/src/java/org/apache/hadoop/hive/ql/udf/generic/NumericHistogram.java ql/src/java/org/apache/hadoop/hive/ql/util/JavaDataModel.java To: JIRA, navis Cc: njain > Size of aggregation buffer which uses non-primitive type is not estimated correctly > ----------------------------------------------------------------------------------- > > Key: HIVE-4068 > URL: https://issues.apache.org/jira/browse/HIVE-4068 > Project: Hive > Issue Type: Improvement > Components: Query Processor > Reporter: Navis > Assignee: Navis > Priority: Minor > Attachments: HIVE-4068.D8859.1.patch, HIVE-4068.D8859.2.patch, HIVE-4068.D8859.3.patch, HIVE-4068.D8859.4.patch > > > Currently, hive assumes an aggregation buffer which holds a map is occupying just 256 byte (fixed). If it's bigger than that in real, OutOfMemoryError can be thrown (especially for >1k buffer). > workaround : set hive.map.aggr.hash.percentmemory= -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira