hive-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Panagiotis Garefalakis (Jira)" <j...@apache.org>
Subject [jira] [Comment Edited] (HIVE-23014) ORC reading performance
Date Thu, 12 Mar 2020 12:34:00 GMT

    [ https://issues.apache.org/jira/browse/HIVE-23014?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17057875#comment-17057875
] 

Panagiotis Garefalakis edited comment on HIVE-23014 at 3/12/20, 12:33 PM:
--------------------------------------------------------------------------

 [~petertoth] thanks for reporting this.

It seems that the particular regression is observed on Single Column Scans >From Wide Columns: [https://github.com/apache/spark/blob/f5118f81e395bde0cd8253dbef6a9e6455c3958a/sql/hive/src/test/scala/org/apache/spark/sql/hive/orc/OrcReadBenchmark.scala#L267]
 Did you observe similar ReaderImpl behaviour in other cases? The only change was Hive version
from 1.2.1 to 2.3.6 ?


was (Author: pgaref):
 [~petertoth] thanks for reporting this.

It seems that the particular regression is observed on Single Column Scans >From Wide Columns: [https://github.com/apache/spark/blob/f5118f81e395bde0cd8253dbef6a9e6455c3958a/sql/hive/src/test/scala/org/apache/spark/sql/hive/orc/OrcReadBenchmark.scala#L267]
Did you observe similar ReaderImpl behaviour in other cases?

> ORC reading performance
> -----------------------
>
>                 Key: HIVE-23014
>                 URL: https://issues.apache.org/jira/browse/HIVE-23014
>             Project: Hive
>          Issue Type: Bug
>          Components: ORC
>    Affects Versions: 2.3.6
>            Reporter: Peter Toth
>            Priority: Major
>
> Spark 3 adds support for using Hive 2.3.6 besides the old Hive 1.2.1 version. Some of
the ORC reading benchmark shows that there is a huge performance difference in ORC reading
between the 2 versions. I measured that {{org.apache.hadoop.hive.ql.io.orc.ReaderImpl}} in
hive-exec-2.3.6-core.jar is ~3-5 times slower than in hive-exec-1.2.1.spark2.jar.
> I'm not sure if more recent Hive versions still suffer from this performance regression.
> Please see some details here: SPARK-30565



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Mime
View raw message