hadoop-mapreduce-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Siddharth Seth (Commented) (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (MAPREDUCE-4099) ApplicationMaster may fail to remove staging directory
Date Tue, 10 Apr 2012 01:47:17 GMT

    [ https://issues.apache.org/jira/browse/MAPREDUCE-4099?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13250387#comment-13250387
] 

Siddharth Seth commented on MAPREDUCE-4099:
-------------------------------------------

bq. Initially I thought a quick fix would be to change MRAppMaster so the call to cleanupStagingDir()
occurs before stopping all the services (and therefore before the RM tries to kill the AM).
However this introduces another problem: if something goes wrong with the AM (killed, crashed,
hung) between the time it has removed the staging directory and before it has notified the
RM then the RM will think the AM did not complete successfully and it will either report the
job as failed (after the AM already told the client it was successful) or worse, the RM will
launch another AM attempt and fail because the staging directory has been removed.

A similar situation would still exist. During service shutdown, all exceptions are ignored
(logged) - and an attempt is made to shutdown all subsequent services. After service stop
- the AM would proceed and delete the staging directory.
A really bad case is if there was an error talking to the RM during the unregister. The client
has already been told that the job is successful - the RM would have no idea, and the AM would
eventually delete the staging directory and exit.

Deleting the staging directory just before the RMCommunicator is stopped would be a much simpler
change. It would however have the same problem in case of a failed unregister. Subsequent
services do not matter.

The HistoryEventHandler had a similar race - where the AM was being shutdown before the history
handler was stopped. That was fixed by ensuring the history service was registered after the
container allocator - so that it shuts down first.

Handling the situation where the client thinks the job is successful, and the RM has no idea
about the job seems like a separate Jira. Some way to have the RM pick up the state of the
job if and when it restarts the next AM.
                
> ApplicationMaster may fail to remove staging directory
> ------------------------------------------------------
>
>                 Key: MAPREDUCE-4099
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4099
>             Project: Hadoop Map/Reduce
>          Issue Type: Bug
>          Components: mrv2
>    Affects Versions: 0.23.2
>            Reporter: Jason Lowe
>            Assignee: Jason Lowe
>            Priority: Critical
>         Attachments: MAPREDUCE-4099.patch, MAPREDUCE-4099.patch
>
>
> When the ApplicationMaster shuts down it's supposed to remove the staging directory,
assuming properties weren't set to override this behavior. During shutdown the AM tells the
ResourceManager that it has finished before it cleans up the staging directory.  However upon
hearing the AM has finished, the RM turns right around and kills the AM container.  If the
AM is too slow, the AM will be killed before the staging directory is removed.
> We're seeing the AM lose this race fairly consistently on our clusters, and the lack
of staging directory cleanup quickly leads to filesystem quota issues for some users.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

Mime
View raw message