hive-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Vihang Karajgaonkar (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HIVE-16024) MSCK Repair Requires nonstrict hive.mapred.mode
Date Fri, 03 Mar 2017 01:03:45 GMT

    [ https://issues.apache.org/jira/browse/HIVE-16024?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15893511#comment-15893511
] 

Vihang Karajgaonkar commented on HIVE-16024:
--------------------------------------------

[~spena] Thats a good point. But I think each PartitionResult object is just a container of
2 String objects (PartitionName and TableName) which is probably more light-weight than Partition
object. It will only be used when metastore knows about it and it not present in FS. That
may be an uncommon case (reverse cases where partition is on FS and not present in Metastore
might be more common). Another idea would be to use {{StringInternUtils.internUriStringsInPath(partPath)}}
like it was done by Misha in HIVE-15882 so that lots of duplicate strings are eliminated.
These path objects might already have been interned when they were discovered or added the
first time but doesn't hurt doing them again here. We should perhaps do some experiments to
see when OOM happens for msck.

> MSCK Repair Requires nonstrict hive.mapred.mode
> -----------------------------------------------
>
>                 Key: HIVE-16024
>                 URL: https://issues.apache.org/jira/browse/HIVE-16024
>             Project: Hive
>          Issue Type: Bug
>          Components: Metastore
>    Affects Versions: 2.2.0
>            Reporter: Barna Zsombor Klara
>            Assignee: Barna Zsombor Klara
>         Attachments: HIVE-16024.01.patch, HIVE-16024.02.patch, HIVE-16024.03.patch, HIVE-16024.04.patch
>
>
> MSCK repair fails when hive.mapred.mode is set to strict
> HIVE-13788 modified the way we read up partitions for a table to improve performance.
Unfortunately it is using PartitionPruner to load the partitions which in turn is checking
hive.mapred.mode.
> The previous code did not check hive.mapred.mode.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

Mime
View raw message