hadoop-mapreduce-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Robert Joseph Evans (Commented) (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (MAPREDUCE-3460) MR AM can hang if containers are allocated on a node blacklisted by the AM
Date Mon, 28 Nov 2011 22:07:40 GMT

    [ https://issues.apache.org/jira/browse/MAPREDUCE-3460?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13158848#comment-13158848
] 

Robert Joseph Evans commented on MAPREDUCE-3460:
------------------------------------------------

+1 to Hitesh's patch at least as a quick fix.  I can try and reproduce the issue here and
verify that the patch does indeed fix the issue.  I can also add in a few unit tests for it
and turn it into a real patch if you like.

I would also like some feedback on a potential (long term) refactor of the code which would
be done on a separate JIRA after 0.23 stabilizes.  It seems to me that the root cause of this
issue is because a special condition for a FAST_FAIL_MAP was missed.  The code right now is
written with lots of if else statements separating out map tasks from reduce tasks and also
from failed map tasks, etc.  I think it would be cleaner to replace the if statements with
classes that use polymorphism to change the methods called.  This would allow the different
handling of a failed map from a normal map or from a reduce to be more evident.  It would
also force the internal data structures that keep track of the different types of tasks to
be combined together.  This is just something that popped into my head while trying to evaluate
Hitesh's fix.  I have not really evaluated what it would take to make it work or anything,
I would just like some feedback about the idea before filing a JIRA a
 bout it.
                
> MR AM can hang if containers are allocated on a node blacklisted by the AM
> --------------------------------------------------------------------------
>
>                 Key: MAPREDUCE-3460
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3460
>             Project: Hadoop Map/Reduce
>          Issue Type: Bug
>          Components: mr-am, mrv2
>    Affects Versions: 0.23.0
>            Reporter: Siddharth Seth
>            Priority: Blocker
>
> When an AM is assigned a FAILED_MAP (priority = 5) container on a nodemanager which it
has blacklisted - it tries to
> find a corresponding container request.
> This uses the hostname to find the matching container request - and can end up returning
any of the ContainerRequests which may have requested a container on this node. This container
request is cleaned to remove the bad node - and then added back to the RM 'ask' list.
> The AM cleans the 'ask' list after each heartbeat - The RM Allocator is still aware of
the priority=5 container (in 'remoteRequestsTable') - but this never gets added back to the
'ask' set - which is what is sent to the RM.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

Mime
View raw message