hadoop-mapreduce-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Vinod Kumar Vavilapalli (Commented) (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (MAPREDUCE-2762) [MR-279] - Cleanup staging dir after job completion
Date Thu, 29 Sep 2011 14:02:45 GMT

    [ https://issues.apache.org/jira/browse/MAPREDUCE-2762?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13117304#comment-13117304
] 

Vinod Kumar Vavilapalli commented on MAPREDUCE-2762:
----------------------------------------------------

This works for most of the cases, but there is a corner condition - if the staging files are
deleted and then AM crashes for whatever reason before it unregisters with the RM, AM restart
is affected. RM will happily restart the AM but the AM cannot recover because of absent job-files.

I think we should do the cleanup as the last step of the MR AM(main() ? ).
                
> [MR-279] - Cleanup staging dir after job completion
> ---------------------------------------------------
>
>                 Key: MAPREDUCE-2762
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2762
>             Project: Hadoop Map/Reduce
>          Issue Type: Bug
>          Components: mrv2
>    Affects Versions: 0.23.0
>            Reporter: Ramya Sunil
>            Assignee: Mahadev konar
>            Priority: Blocker
>             Fix For: 0.23.0
>
>         Attachments: MAPREDUCE-2762.patch, MAPREDUCE-2762.patch, MAPREDUCE-2762.patch
>
>
> The files created under the staging dir have to be deleted after job completion. Currently,
all job.* files remain forever in the ${yarn.apps.stagingDir}

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

Mime
View raw message