hive-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Mostafa Mokhtar (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (HIVE-8291) ACID : Reading from partitioned bucketed tables has high overhead, 50% of time is spent in OrcInputFormat.getReader
Date Mon, 29 Sep 2014 18:57:34 GMT

     [ https://issues.apache.org/jira/browse/HIVE-8291?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Mostafa Mokhtar updated HIVE-8291:
----------------------------------
    Attachment: 2014_09_28_16_48_48.jfr

Hot function profile.
Use Java mission control (jmc) to open the file, JMC is part of Java 7.

> ACID : Reading from partitioned bucketed tables has high overhead, 50% of time is spent
in OrcInputFormat.getReader
> -------------------------------------------------------------------------------------------------------------------
>
>                 Key: HIVE-8291
>                 URL: https://issues.apache.org/jira/browse/HIVE-8291
>             Project: Hive
>          Issue Type: Bug
>          Components: Query Processor
>    Affects Versions: 0.14.0
>         Environment: cn105
>            Reporter: Mostafa Mokhtar
>            Assignee: Owen O'Malley
>             Fix For: 0.14.0
>
>         Attachments: 2014_09_28_16_48_48.jfr
>
>
> Reading from bucketed partitioned tables has significantly higher overhead compared to
non-bucketed non-partitioned files.
> 50% of the time is spent in these two lines of code in OrcInputFormate.getReader()
> {code}
>     String txnString = conf.get(ValidTxnList.VALID_TXNS_KEY,
>                                 Long.MAX_VALUE + ":");
>     ValidTxnList validTxnList = new ValidTxnListImpl(txnString);
> {code}
> {code}
> Stack Trace	Sample Count	Percentage(%)
> hive.ql.exec.tez.MapRecordSource.pushRecord()	2,981	87.215
> 	org.apache.tez.mapreduce.lib.MRReaderMapred.next()	2,002	58.572
>     	mapred.split.TezGroupedSplitsInputFormat$TezGroupedSplitsRecordReader.next(Object,
Object)	2,002	58.572
> 			mapred.split.TezGroupedSplitsInputFormat$TezGroupedSplitsRecordReader.initNextRecordReader()
1,984	58.046
>             	hive.ql.io.HiveInputFormat.getRecordReader(InputSplit, JobConf, Reporter)
1,983	58.016
>                 	hive.ql.io.orc.OrcInputFormat.getRecordReader(InputSplit, JobConf, Reporter)
1,891	55.325
>                     	hive.ql.io.orc.OrcInputFormat.getReader(InputSplit, AcidInputFormat$Options)
1,723	50.41
>                         	hive.common.ValidTxnListImpl.<init>(String)	934	27.326
>                             conf.Configuration.get(String, String)	621	18.169
>  {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message