hive-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Ratheesh Kamoor (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HIVE-14925) MSCK repair table hang while running with multi threading enabled
Date Thu, 13 Oct 2016 00:12:20 GMT

    [ https://issues.apache.org/jira/browse/HIVE-14925?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15570328#comment-15570328
] 

Ratheesh Kamoor commented on HIVE-14925:
----------------------------------------

Are you trying with partitions in hdfs? You may not run into issues if threads are fast enough
to finish execution before recursive call happens, File systems like S3 will clearly shows
error due to n/w latency. 

> MSCK repair table hang while running with multi threading enabled
> -----------------------------------------------------------------
>
>                 Key: HIVE-14925
>                 URL: https://issues.apache.org/jira/browse/HIVE-14925
>             Project: Hive
>          Issue Type: Bug
>          Components: CLI
>    Affects Versions: 2.2.0
>            Reporter: Ratheesh Kamoor
>            Assignee: Ratheesh Kamoor
>            Priority: Critical
>             Fix For: 2.2.0
>
>         Attachments: HIVE-14925.patch
>
>
> MSCK REPAIR TABLE hanging while running with multi-threading enabled (default). I think
it is because of a major design flaw in how thread pool implemented in HiveMetaSoreChecker
class / checkPartitionDirs method. This method has a thread pool which register Callable but
callable makes a recursive call to checkPartitionDirs method again. This code will hang when
number of directories is more than thread pool size. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message