Mailing-List: contact issues-help@hive.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@hive.apache.org
Date: Thu, 30 Mar 2017 06:59:41 +0000 (UTC)
From: "Dudu Markovitz (JIRA)" <jira@apache.org>
To: issues@hive.apache.org
Message-ID: <JIRA.13059240.1490559413000.162237.1490857181566@Atlassian.JIRA>
In-Reply-To: <JIRA.13059240.1490559413000@Atlassian.JIRA>
References: <JIRA.13059240.1490559413000@Atlassian.JIRA> <JIRA.13059240.1490559413139@jira-lw-us.apache.org>
Subject: [jira] [Commented] (HIVE-16299) In case of partitioned table, MSCK
 REPAIR TABLE does not do a full validation of a FS paths and in result
 create false partitions and directories
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
archived-at: Thu, 30 Mar 2017 06:59:46 -0000


    [ https://issues.apache.org/jira/browse/HIVE-16299?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15948517#comment-15948517 ] 

Dudu Markovitz commented on HIVE-16299:
---------------------------------------

Hi

1.
The 1st example could be created on hive 1.1.0-cdh5.7.0 but not on Hive 1.2.1000.2.5.3.0-37.
The 2nd example could be created on both.

2.
I think that a situation where a partition is created built on directories structure and then directories are built based on the partitions that were just added, is not healthy and should be prevented.
It would make a lot of sense to validated a complete path and not each directory separately.


> In case of partitioned table, MSCK REPAIR TABLE does not do a full validation of a FS paths and in result create false partitions and directories
> -------------------------------------------------------------------------------------------------------------------------------------------------
>
>                 Key: HIVE-16299
>                 URL: https://issues.apache.org/jira/browse/HIVE-16299
>             Project: Hive
>          Issue Type: Bug
>          Components: Metastore
>    Affects Versions: storage-2.2.0
>            Reporter: Dudu Markovitz
>            Assignee: Vihang Karajgaonkar
>            Priority: Minor
>
> https://github.com/apache/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/metadata/HiveMetaStoreChecker.java
> static String getPartitionName(Path tablePath, Path partitionPath, Set<String> partCols)
> ------------------------------------------------------------------------------------
> MSCK REPAIR validates that any sub-directory is in the format col=val and that there is indeed a partition column named "col".
> However, there is no validation of the partition column location and as a result false partitions are being created and so are directories that match those partitions. 
> e.g. 1
> hive> dfs -mkdir -p /user/hive/warehouse/t/a=1/a=2/a=3/b=4/c=5;
> hive> create external table t (i int) partitioned by (a int,b int,c int) ;
> OK
> hive> msck repair table t;
> OK
> Partitions not in metastore:	t:a=1/a=2/a=3/b=4/c=5
> Repair: Added partition to metastore t:a=1/a=2/a=3/b=4/c=5
> Time taken: 0.563 seconds, Fetched: 2 row(s)
> hive> show partitions t;
> OK
> a=3/b=4/c=5
> hive> dfs -ls -R /user/hive/warehouse/t;
> drwxr-xr-x   - cloudera supergroup          0 2017-03-26 13:07 /user/hive/warehouse/t/a=1
> drwxr-xr-x   - cloudera supergroup          0 2017-03-26 13:07 /user/hive/warehouse/t/a=1/a=2
> drwxr-xr-x   - cloudera supergroup          0 2017-03-26 13:07 /user/hive/warehouse/t/a=1/a=2/a=3
> drwxr-xr-x   - cloudera supergroup          0 2017-03-26 13:07 /user/hive/warehouse/t/a=1/a=2/a=3/b=4
> drwxr-xr-x   - cloudera supergroup          0 2017-03-26 13:07 /user/hive/warehouse/t/a=1/a=2/a=3/b=4/c=5
> drwxrwxrwx   - cloudera supergroup          0 2017-03-26 13:07 /user/hive/warehouse/t/a=3
> drwxrwxrwx   - cloudera supergroup          0 2017-03-26 13:07 /user/hive/warehouse/t/a=3/b=4
> drwxrwxrwx   - cloudera supergroup          0 2017-03-26 13:07 /user/hive/warehouse/t/a=3/b=4/c=5
> e.g. 2
> hive> dfs -mkdir -p /user/hive/warehouse/t/c=3/b=2/a=1;
> hive> create external table t (i int) partitioned by (a int,b int,c int);
> OK
> hive> msck repair table t;
> OK
> Partitions not in metastore:	t:c=3/b=2/a=1
> Repair: Added partition to metastore t:c=3/b=2/a=1
> Time taken: 0.512 seconds, Fetched: 2 row(s)
> hive> show partitions t;
> OK
> a=1/b=2/c=3
> hive> dfs -ls -R  /user/hive/warehouse/t;
> drwxrwxrwx   - cloudera supergroup          0 2017-03-26 13:13 /user/hive/warehouse/t/a=1
> drwxrwxrwx   - cloudera supergroup          0 2017-03-26 13:13 /user/hive/warehouse/t/a=1/b=2
> drwxrwxrwx   - cloudera supergroup          0 2017-03-26 13:13 /user/hive/warehouse/t/a=1/b=2/c=3
> drwxr-xr-x   - cloudera supergroup          0 2017-03-26 13:12 /user/hive/warehouse/t/c=3
> drwxr-xr-x   - cloudera supergroup          0 2017-03-26 13:12 /user/hive/warehouse/t/c=3/b=2
> drwxr-xr-x   - cloudera supergroup          0 2017-03-26 13:12 /user/hive/warehouse/t/c=3/b=2/a=1


--
This message was sent by Atlassian JIRA
(v6.3.15#6346)