Mailing-List: contact yarn-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
Date: Mon, 20 Nov 2017 06:34:00 +0000 (UTC)
From: "YunFan Zhou (JIRA)" <jira@apache.org>
To: yarn-issues@hadoop.apache.org
Message-ID: <JIRA.13119552.1511150947000.277643.1511159640128@Atlassian.JIRA>
In-Reply-To: <JIRA.13119552.1511150947000@Atlassian.JIRA>
References: <JIRA.13119552.1511150947000@Atlassian.JIRA> <JIRA.13119552.1511150947445@jira-lw-us.apache.org>
Subject: [jira] [Comment Edited] (YARN-7534) Fair scheduler assign resources
 may exceed maxResources
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
archived-at: Mon, 20 Nov 2017 06:34:06 -0000


    [ https://issues.apache.org/jira/browse/YARN-7534?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16258849#comment-16258849 ] 

YunFan Zhou edited comment on YARN-7534 at 11/20/17 6:33 AM:
-------------------------------------------------------------

[~templedf]
For example, a queue resource usage time slices as follows:
Max Resources: *<memory:10, vCores:10>*
Current used resources:	*<memory:9, vCores:9>*
Pending resource request: *<memory:2, vCores:2>*

This time a node manager report the heartbeat and it has *<memory:3, vCores:3>* available resources.
Before assigning containers it will do follows check:
{code:java}
@Override
  public Resource assignContainer(FSSchedulerNode node) {
    Resource assigned = Resources.none();
    if (LOG.isDebugEnabled()) {
      LOG.debug("Node " + node.getNodeName() + " offered to queue: " +
          getName() + " fairShare: " + getFairShare());
    }

    if (!assignContainerPreCheck(node)) {
      return assigned;
    }
{code}

Because it used resources is less than maxResources. So it will assign *<memory:2, vCores:2>* to
this queue, and in this time the queue's used resources exceed *maxResources*.


was (Author: daemon):
[~templedf]
For example, a queue resource usage time slices as follows:
Max Resources: *<memory:10, vCores:10>*
Current used resources:	*<memory:9, vCores:9>*
Pending resource request: *<memory:2, vCores:2>*

This time a node manager report the heartbeat and it has *<memory:3, vCores:3>* available resources.
Before assigning containers it will do follows check:
{code:java}
@Override
  public Resource assignContainer(FSSchedulerNode node) {
    Resource assigned = Resources.none();
    if (LOG.isDebugEnabled()) {
      LOG.debug("Node " + node.getNodeName() + " offered to queue: " +
          getName() + " fairShare: " + getFairShare());
    }

    if (!assignContainerPreCheck(node)) {
      return assigned;
    }
{code}

Because it used resources is less than maxResources. So it will assign *<memory:2, vCores:2>* to
this queue, and in this time the queue's used resources over limit.


> Fair scheduler assign resources may exceed maxResources
> -------------------------------------------------------
>
>                 Key: YARN-7534
>                 URL: https://issues.apache.org/jira/browse/YARN-7534
>             Project: Hadoop YARN
>          Issue Type: Bug
>          Components: fairscheduler
>            Reporter: YunFan Zhou
>
> The logic we're scheduling now is to check whether the resources used by the queue has exceeded *maxResources* before assigning the container. This will leads to the fact that after assigning this container the queue uses more resources than *maxResources*.


--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

---------------------------------------------------------------------
To unsubscribe, e-mail: yarn-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: yarn-issues-help@hadoop.apache.org