hadoop-mapreduce-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Vinod Kumar Vavilapalli (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (MAPREDUCE-4751) AM stuck in KILL_WAIT for days
Date Sat, 10 Nov 2012 02:07:12 GMT

    [ https://issues.apache.org/jira/browse/MAPREDUCE-4751?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13494518#comment-13494518
] 

Vinod Kumar Vavilapalli commented on MAPREDUCE-4751:
----------------------------------------------------

bq. Part of the issue is that the job is hanging around waiting for all tasks to be killed
rather than just exiting and letting YARN shoot any straggling containers. I think it would
be simpler/safer for the AM to just write out the final state stuff and exit, much like it
does for the FAILED state. If job's KILL_WAIT really is necessary then we'd need a corresponding
FAILED_WAIT state to handle waiting for task cleanup when a job fails.
I agree. Sharad/I debated this for a while when we wrote this initially. We let it be like
it is now, just to be sure that AM's sanely exit, but we can change it. The only catch I can
think of is, while the AM tries to do the remaining cleanup work (jobhistory etc), tasks will
keep on bombarding AM with more updates.

Didn't realize that we don't have fail_wait state.

The change isn't much bigger but it can break tests. Let's pursue that separately.

The current bug is caused by Tasks waiting on TAs which should be fixed by my patch. Of course,
it then opens up the job bug, let's fix that separately.

Regarding doing away with Task's kill_wait, I disagree. Tasks can get kill signal during the
AM is running, so we should handle it explicitly by killing and waiting for all attempts,
otherwise we run the risk of dangling JVMs doing nothing but occupying slots till AM exits.
                
> AM stuck in KILL_WAIT for days
> ------------------------------
>
>                 Key: MAPREDUCE-4751
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4751
>             Project: Hadoop Map/Reduce
>          Issue Type: Bug
>    Affects Versions: 0.23.3, 2.0.2-alpha
>            Reporter: Ravi Prakash
>            Assignee: Vinod Kumar Vavilapalli
>         Attachments: MAPREDUCE-4751-20121108.txt, MAPREDUCE-4751-20121109.txt, TaskAttemptStateGraph.jpg
>
>
> We found some jobs were stuck in KILL_WAIT for days on end. The RM shows them as RUNNING.
When you go to the AM, it shows it in the KILL_WAIT state, and a few maps running. All these
maps were scheduled on nodes which are now in the RM's Lost nodes list. The running maps are
in the FAIL_CONTAINER_CLEANUP state

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

Mime
View raw message