hive-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Gang Tim Liu (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (HIVE-3251) Hive doesn't remove scrach directories while killing running MR job
Date Wed, 11 Jul 2012 23:07:35 GMT

     [ https://issues.apache.org/jira/browse/HIVE-3251?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Gang Tim Liu updated HIVE-3251:
-------------------------------

    Attachment: HIVE-3151.patch
    
> Hive doesn't remove scrach directories while killing running MR job
> -------------------------------------------------------------------
>
>                 Key: HIVE-3251
>                 URL: https://issues.apache.org/jira/browse/HIVE-3251
>             Project: Hive
>          Issue Type: Bug
>          Components: Server Infrastructure
>            Reporter: Gang Tim Liu
>            Assignee: Gang Tim Liu
>         Attachments: HIVE-3151.patch
>
>
> While killing running MR job, hive doesn't clean up scratch directory (mapred.cache.files).
So that, afterwards, scratch directory is left there in hdfs. HDFS name node doesn't know
it and try to do lease recovery. while such instances happen more, it will eventually crash
namenode.
> The fix is to leverage hdfs clean up functionality. While creating scratch dirs, hive
registers it to hdfs cleanup hook. While killing happens, hdfs will clean them up.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

Mime
View raw message