hive-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Chris Nauroth (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HIVE-14739) Replace runnables directly added to runtime shutdown hooks to avoid deadlock
Date Mon, 12 Sep 2016 22:30:20 GMT

    [ https://issues.apache.org/jira/browse/HIVE-14739?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15485485#comment-15485485
] 

Chris Nauroth commented on HIVE-14739:
--------------------------------------

[~prasanth_j], thank you for sharing this patch.  It's interesting for me to see that Hive
appears to have forked its own copy of {{ShutdownHookManager}} from Hadoop.  I don't know
the background on this.  The code is similar, but not identical, between the two codebases.
 Perhaps that's because the Hive version was not updated to match recent changes in Hadoop,
like HADOOP-12950.

In order to fully prevent deadlocks between different shutdown hooks, there really needs to
be a single {{ShutdownHookManager}} in the process.  If Hadoop and Hive each have their own
implementation, and a Hive process instantiates one of each and registers different shutdown
hooks with each one, then there will be 2 threads executing different shutdown hooks concurrently,
which could still cause a deadlock.

Would it make sense to eliminate the forked {{ShutdownHookManager}} class in Hive and instead
rely completely on using the one from Hadoop?

Also, a minor nit: maybe all calls to {{new Thread()}} could be converted to {{new Runnable()}}.
 The {{Runnable}} interface is sufficient, and it won't make use of any additional functionality
provided by the {{Thread}} implementation.

> Replace runnables directly added to runtime shutdown hooks to avoid deadlock
> ----------------------------------------------------------------------------
>
>                 Key: HIVE-14739
>                 URL: https://issues.apache.org/jira/browse/HIVE-14739
>             Project: Hive
>          Issue Type: Bug
>    Affects Versions: 2.2.0
>            Reporter: Deepesh Khandelwal
>            Assignee: Prasanth Jayachandran
>         Attachments: HIVE-14739.1.patch
>
>
> [~deepesh] reported that a deadlock can occur when running queries through hive cli.
[~cnauroth] analyzed it and reported that hive adds shutdown hooks directly to java Runtime
which may execute in non-deterministic order causing deadlocks with hadoop's shutdown hooks.
In one case, hadoop shutdown locked FileSystem#Cache and FileSystem.close whereas hive shutdown
hook locked FileSystem.close and FileSystem#Cache order causing a deadlock. 
> Hive and Hadoop has ShutdownHookManager that runs the shutdown hooks in deterministic
order based on priority. We should use that to avoid deadlock throughout the code.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message