Return-Path: 
 <dev-return-105403-apmail-hive-dev-archive=hive.apache.org@hive.apache.org>
X-Original-To: apmail-hive-dev-archive@www.apache.org
Delivered-To: apmail-hive-dev-archive@www.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id C02E317D50
	for <apmail-hive-dev-archive@www.apache.org>;
 Mon, 29 Sep 2014 18:54:37 +0000 (UTC)
Received: (qmail 14189 invoked by uid 500); 29 Sep 2014 18:54:35 -0000
Delivered-To: apmail-hive-dev-archive@hive.apache.org
Received: (qmail 14133 invoked by uid 500); 29 Sep 2014 18:54:35 -0000
Mailing-List: contact dev-help@hive.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:dev-help@hive.apache.org>
List-Unsubscribe: <mailto:dev-unsubscribe@hive.apache.org>
List-Post: <mailto:dev@hive.apache.org>
List-Id: <dev.hive.apache.org>
Reply-To: dev@hive.apache.org
Delivered-To: mailing list dev@hive.apache.org
Received: (qmail 14088 invoked by uid 500); 29 Sep 2014 18:54:35 -0000
Delivered-To: apmail-hadoop-hive-dev@hadoop.apache.org
Received: (qmail 14078 invoked by uid 99); 29 Sep 2014 18:54:35 -0000
Received: from arcas.apache.org (HELO arcas.apache.org) (140.211.11.28)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 29 Sep 2014 18:54:35 +0000
Date: Mon, 29 Sep 2014 18:54:34 +0000 (UTC)
From: "Mostafa Mokhtar (JIRA)" <jira@apache.org>
To: hive-dev@hadoop.apache.org
Message-ID: <JIRA.12744750.1412016755000.147503.1412016874982@Atlassian.JIRA>
In-Reply-To: <JIRA.12744750.1412016755000@Atlassian.JIRA>
References: <JIRA.12744750.1412016755000@Atlassian.JIRA>
 <JIRA.12744750.1412016755268@arcas>
Subject: [jira] [Updated] (HIVE-8292) Reading from partitioned bucketed
 tables has high overhead in MapOperator.cleanUpInputFileChangedOp
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HIVE-8292?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Mostafa Mokhtar updated HIVE-8292:
----------------------------------
    Attachment: 2014_09_29_14_46_04.jfr

Hot function profile

> Reading from partitioned bucketed tables has high overhead in MapOperator.cleanUpInputFileChangedOp
> ---------------------------------------------------------------------------------------------------
>
>                 Key: HIVE-8292
>                 URL: https://issues.apache.org/jira/browse/HIVE-8292
>             Project: Hive
>          Issue Type: Bug
>          Components: Query Processor
>    Affects Versions: 0.14.0
>         Environment: cn105
>            Reporter: Mostafa Mokhtar
>            Assignee: Prasanth J
>             Fix For: 0.14.0
>
>         Attachments: 2014_09_29_14_46_04.jfr
>
>
> Reading from bucketed partitioned tables has significantly higher overhead compared to non-bucketed non-partitioned files.
> 20% of the profile is spent in MapOperator.cleanUpInputFileChangedOp
> 5% the CPU in 
> {code}
>  Path onepath = normalizePath(onefile);
> {code}
> And 
> 15% the CPU in 
> {code}
>  onepath.toUri().relativize(fpath.toUri()).equals(fpath.toUri());
> {code}
> From the profiler 
> {code}
> Stack Trace	Sample Count	Percentage(%)
> org.apache.hadoop.hive.ql.exec.tez.MapRecordSource.processRow(Object)	978	28.613
>    org.apache.hadoop.hive.ql.exec.vector.VectorMapOperator.process(Writable)	978	28.613
>       org.apache.hadoop.hive.ql.exec.Operator.cleanUpInputFileChanged()	866	25.336
>          org.apache.hadoop.hive.ql.exec.MapOperator.cleanUpInputFileChangedOp()	866	25.336
>             java.net.URI.relativize(URI)	655	19.163
>                java.net.URI.relativize(URI, URI)	655	19.163
>                   java.net.URI.normalize(String)	517	15.126
> 					java.net.URI.needsNormalization(String)	372	10.884
> 					   java.lang.String.charAt(int)	235	6.875
> 									  java.net.URI.equal(String, String)	27	0.79
> 									  java.lang.StringBuilder.toString()	1	0.029
> 									  java.lang.StringBuilder.<init>()	1	0.029
> 									  java.lang.StringBuilder.append(String)	1	0.029
> 								org.apache.hadoop.hive.ql.exec.MapOperator.normalizePath(String)	167	4.886
> 								   org.apache.hadoop.fs.Path.<init>(String)	162	4.74
> 									  org.apache.hadoop.fs.Path.initialize(String, String, String, String)	162	4.74
> 	org.apache.hadoop.fs.Path.normalizePath(String, String)	97	2.838
> 	   org.apache.commons.lang.StringUtils.replace(String, String, String)	97	2.838
> 		  org.apache.commons.lang.StringUtils.replace(String, String, String, int)	97	2.838
> 			 java.lang.String.indexOf(String, int)	97	2.838
> 		java.net.URI.<init>(String, String, String, String, String)	65	1.902
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)