Return-Path: X-Original-To: apmail-hive-dev-archive@www.apache.org Delivered-To: apmail-hive-dev-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 8B91A11AF5 for ; Tue, 5 Aug 2014 16:22:12 +0000 (UTC) Received: (qmail 47251 invoked by uid 500); 5 Aug 2014 16:22:12 -0000 Delivered-To: apmail-hive-dev-archive@hive.apache.org Received: (qmail 47181 invoked by uid 500); 5 Aug 2014 16:22:12 -0000 Mailing-List: contact dev-help@hive.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@hive.apache.org Delivered-To: mailing list dev@hive.apache.org Received: (qmail 47169 invoked by uid 500); 5 Aug 2014 16:22:12 -0000 Delivered-To: apmail-hadoop-hive-dev@hadoop.apache.org Received: (qmail 47166 invoked by uid 99); 5 Aug 2014 16:22:12 -0000 Received: from arcas.apache.org (HELO arcas.apache.org) (140.211.11.28) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 05 Aug 2014 16:22:12 +0000 Date: Tue, 5 Aug 2014 16:22:12 +0000 (UTC) From: "Mostafa Mokhtar (JIRA)" To: hive-dev@hadoop.apache.org Message-ID: In-Reply-To: References: Subject: [jira] [Updated] (HIVE-7574) CommonJoinOperator.checkAndGenObject calls LOG.Trace per row from probe side in a HashMap join consuming 4% of the CPU MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394 [ https://issues.apache.org/jira/browse/HIVE-7574?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mostafa Mokhtar updated HIVE-7574: ---------------------------------- Attachment: HIVE-7574.1.patch Patch with fix. > CommonJoinOperator.checkAndGenObject calls LOG.Trace per row from probe side in a HashMap join consuming 4% of the CPU > ---------------------------------------------------------------------------------------------------------------------- > > Key: HIVE-7574 > URL: https://issues.apache.org/jira/browse/HIVE-7574 > Project: Hive > Issue Type: Bug > Components: Query Processor > Affects Versions: 0.13.1 > Reporter: Mostafa Mokhtar > Assignee: Mostafa Mokhtar > Labels: perfomance > Fix For: 0.14.0 > > Attachments: HIVE-7574.1.patch > > Original Estimate: 2h > Remaining Estimate: 2h > > In Map join Log4JLogger.trace takes 4% of the CPU time as it gets called per row from the probe side by CommonJoinOperator.genAllOneUniqueJoinObject. > Fix is to remove the logging code code below from CommonJoinOperator.genAllOneUniqueJoinObject: > {code} > if (allOne) { > LOG.info("calling genAllOneUniqueJoinObject"); > genAllOneUniqueJoinObject(); > LOG.info("called genAllOneUniqueJoinObject"); > } else { > LOG.trace("calling genUniqueJoinObject"); > genUniqueJoinObject(0, 0); > LOG.trace("called genUniqueJoinObject"); > } > {code} > And > {code} > if (!hasEmpty && !mayHasMoreThanOne) { > LOG.trace("calling genAllOneUniqueJoinObject"); > genAllOneUniqueJoinObject(); > LOG.trace("called genAllOneUniqueJoinObject"); > } else if (!hasEmpty && !hasLeftSemiJoin) { > LOG.trace("calling genUniqueJoinObject"); > genUniqueJoinObject(0, 0); > LOG.trace("called genUniqueJoinObject"); > } else { > LOG.trace("calling genObject"); > genJoinObject(); > LOG.trace("called genObject"); > } > {code} > This is the call stack > {code} > Stack Trace Sample Count Percentage(%) > hadoop.hive.ql.exec.MapJoinOperator.processOp(Object, int) 388 75.486 > hadoop.hive.ql.exec.CommonJoinOperator.checkAndGenObject() 121 23.541 > hadoop.hive.ql.exec.CommonJoinOperator.genAllOneUniqueJoinObject() 92 17.899 > commons.logging.impl.Log4JLogger.trace(Object) 20 3.891 > log4j.Category.log(String, Priority, Object, Throwable) 20 3.891 > log4j.Category.getEffectiveLevel() 10 1.946 > {code} -- This message was sent by Atlassian JIRA (v6.2#6252)