hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Bikas Saha (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (YARN-389) Infinitely assigning containers when the required resource exceeds the cluster's absolute capacity
Date Fri, 22 Mar 2013 18:29:16 GMT

    [ https://issues.apache.org/jira/browse/YARN-389?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13611037#comment-13611037
] 

Bikas Saha commented on YARN-389:
---------------------------------

bq. I've checked reason why the requested AM size is larger than yarn.scheduler.maximum-allocation-mb.
In fact, the AM size is not checked before requesting a container. It is because MR jobs are
submitted through ResourceMgrDelegate and managed by MRAppMaster, while checking AM size against
the maximum resource capability is only implemented in Client and ApplicationMaster of the
distributed shell.

The AM launcher checking limits is great but its the RM's responsibility to not accept requests
that it cannot fulfill. From reading the comments, it not clear to me what exactly the root
issue is in the RM itself. Is it that the RM is accepting container requests that are greater
than the maximum resource available on any 1 node? I dont think we should be comparing against
the entire cluster resource since a container request needs to be satisfied within a single
node.
                
> Infinitely assigning containers when the required resource exceeds the cluster's absolute
capacity
> --------------------------------------------------------------------------------------------------
>
>                 Key: YARN-389
>                 URL: https://issues.apache.org/jira/browse/YARN-389
>             Project: Hadoop YARN
>          Issue Type: Bug
>            Reporter: Zhijie Shen
>            Assignee: Zhijie Shen
>
> I've run wordcount example on branch-2 and trunk. I've set yarn.nodemanager.resource.memory-mb
to 1G and yarn.app.mapreduce.am.resource.mb to 1.5G. Therefore, resourcemanager is to assign
a 2G AM container for AM. However, the nodemanager doesn't have enough memory to assign the
container. The problem is that the assignment operation will be repeated infinitely, if the
assignment cannot be accomplished. Logs follow.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

Mime
View raw message