hive-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Colin Ma (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HIVE-16969) Improvement performance of MapOperator for Parquet
Date Tue, 27 Jun 2017 03:15:00 GMT

    [ https://issues.apache.org/jira/browse/HIVE-16969?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16064191#comment-16064191
] 

Colin Ma commented on HIVE-16969:
---------------------------------

[~csun], [~Ferd], can you help to review the patch? Thanks.

> Improvement performance of MapOperator for Parquet
> --------------------------------------------------
>
>                 Key: HIVE-16969
>                 URL: https://issues.apache.org/jira/browse/HIVE-16969
>             Project: Hive
>          Issue Type: Improvement
>    Affects Versions: 3.0.0
>            Reporter: Colin Ma
>            Assignee: Colin Ma
>             Fix For: 3.0.0
>
>         Attachments: HIVE-16969.001.patch
>
>
> For a table with many partition files, MapOperator.cloneConfsForNestedColPruning() will
update the hive.io.file.readNestedColumn.paths many times. The larger value of hive.io.file.readNestedColumn.paths
will cause the poor performance for ParquetHiveSerDe.processRawPrunedPaths(). 
> So, the unnecessary paths should be appended to hive.io.file.readNestedColumn.paths.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Mime
View raw message