hive-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Eugene Koifman (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HIVE-15202) Concurrent compactions for the same partition may generate malformed folder structure
Date Wed, 16 Nov 2016 17:55:59 GMT

    [ https://issues.apache.org/jira/browse/HIVE-15202?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15671134#comment-15671134
] 

Eugene Koifman commented on HIVE-15202:
---------------------------------------

When compactions are triggered by the system automatically they have logic to avoid concurrent
invocations but not when triggered by the user.  It should and will be fixed (almost certainly
before 2.2 is released)

> Concurrent compactions for the same partition may generate malformed folder structure
> -------------------------------------------------------------------------------------
>
>                 Key: HIVE-15202
>                 URL: https://issues.apache.org/jira/browse/HIVE-15202
>             Project: Hive
>          Issue Type: Bug
>            Reporter: Rui Li
>
> If two compactions run concurrently on a single partition, it may generate folder structure
like this: (nested base dir)
> {noformat}
> drwxr-xr-x   - root supergroup          0 2016-11-14 22:23 /user/hive/warehouse/test/z=1/base_0000007/base_0000007
> -rw-r--r--   3 root supergroup        201 2016-11-14 21:46 /user/hive/warehouse/test/z=1/base_0000007/bucket_00000
> -rw-r--r--   3 root supergroup        611 2016-11-14 21:46 /user/hive/warehouse/test/z=1/base_0000007/bucket_00001
> -rw-r--r--   3 root supergroup        614 2016-11-14 21:46 /user/hive/warehouse/test/z=1/base_0000007/bucket_00002
> -rw-r--r--   3 root supergroup        621 2016-11-14 21:46 /user/hive/warehouse/test/z=1/base_0000007/bucket_00003
> -rw-r--r--   3 root supergroup        621 2016-11-14 21:46 /user/hive/warehouse/test/z=1/base_0000007/bucket_00004
> -rw-r--r--   3 root supergroup        201 2016-11-14 21:46 /user/hive/warehouse/test/z=1/base_0000007/bucket_00005
> -rw-r--r--   3 root supergroup        201 2016-11-14 21:46 /user/hive/warehouse/test/z=1/base_0000007/bucket_00006
> -rw-r--r--   3 root supergroup        201 2016-11-14 21:46 /user/hive/warehouse/test/z=1/base_0000007/bucket_00007
> -rw-r--r--   3 root supergroup        201 2016-11-14 21:46 /user/hive/warehouse/test/z=1/base_0000007/bucket_00008
> -rw-r--r--   3 root supergroup        201 2016-11-14 21:46 /user/hive/warehouse/test/z=1/base_0000007/bucket_00009
> {noformat}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message