hive-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Subramanyam Pattipaka (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HIVE-14511) Improve MSCK for partitioned table to deal with special cases
Date Mon, 15 Aug 2016 17:00:24 GMT

    [ https://issues.apache.org/jira/browse/HIVE-14511?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15421268#comment-15421268
] 

Subramanyam Pattipaka commented on HIVE-14511:
----------------------------------------------

I mean to add only p1=1/p2=1. For example if you have following structure

data/p1=1/p2=1/p3=1
              /p3=2
              /p3=3
         /p2=2/p3=1
              /p3=2
    /p1=2/p2=1/p3=1

Now, I want to add only (1,1), (1,2) and (2,1) as partitions. If you remove the above check
then this is possible.

In first iteration you would list 

p1=1
p1=2

in next iteration you would list 

/p1=1/p2=1
/p1=1/p2=3
/p1=2/p2=1

As depth is 0 we stop here and these are the paths for partitions if user want to create on
p1 and p2 as partition columns. If you want you can check for existence of use of config mapred.input.dir.recursive
and hive.mapred.supports.subdirectories.

> Improve MSCK for partitioned table to deal with special cases
> -------------------------------------------------------------
>
>                 Key: HIVE-14511
>                 URL: https://issues.apache.org/jira/browse/HIVE-14511
>             Project: Hive
>          Issue Type: Sub-task
>            Reporter: Pengcheng Xiong
>            Assignee: Pengcheng Xiong
>         Attachments: HIVE-14511.01.patch
>
>
> Some users will have a folder rather than a file under the last partition folder. However,
msck is going to search for the leaf folder rather than the last partition folder. We need
to improve that.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message