Mailing-List: contact yarn-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
Date: Fri, 4 Aug 2017 08:10:00 +0000 (UTC)
From: "Yufei Gu (JIRA)" <jira@apache.org>
To: yarn-issues@hadoop.apache.org
Message-ID: <JIRA.13092384.1501830084000.92099.1501834200548@Atlassian.JIRA>
In-Reply-To: <JIRA.13092384.1501830084000@Atlassian.JIRA>
References: <JIRA.13092384.1501830084000@Atlassian.JIRA> <JIRA.13092384.1501830084014@jira-lw-us.apache.org>
Subject: [jira] [Resolved] (YARN-6947) The implementation of
 Schedulable#getResourceUsage so inefficiency that can reduce the
 performance of scheduling
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
archived-at: Fri, 04 Aug 2017 08:10:08 -0000


     [ https://issues.apache.org/jira/browse/YARN-6947?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Yufei Gu resolved YARN-6947.
----------------------------
    Resolution: Duplicate

> The implementation of Schedulable#getResourceUsage so inefficiency that can reduce the performance of scheduling
> ----------------------------------------------------------------------------------------------------------------
>
>                 Key: YARN-6947
>                 URL: https://issues.apache.org/jira/browse/YARN-6947
>             Project: Hadoop YARN
>          Issue Type: Improvement
>          Components: fairscheduler
>            Reporter: YunFan Zhou
>            Priority: Critical
>
> Each time the FairScheduler assign container, it will checks whether the resources used by the queue exceed Max Share. However, our current calculation of the resources of the queue is particularly inefficient, which recursively iterates over all child nodes, with high time complexity. 
> We can refactor this logic by using lazy update way.
> {code:java}
>   @Override
>   public Resource assignContainer(FSSchedulerNode node) {
>     Resource assigned = Resources.none();
>     // If this queue is over its limit, reject
>     if (!assignContainerPreCheck(node)) {
>       return assigned;
>     }
> {code}
> {code:java}
>    * Helper method to check if the queue should attempt assigning resources
>    * 
>    * @return true if check passes (can assign) or false otherwise
>    */
>   boolean assignContainerPreCheck(FSSchedulerNode node) {
>     if (node.getReservedContainer() != null) {
>       if (LOG.isDebugEnabled()) {
>         LOG.debug("Assigning container failed on node '" + node.getNodeName()
>             + " because it has reserved containers.");
>       }
>       return false;
>     } else if (!Resources.fitsIn(getResourceUsage(), maxShare)) {
>       if (LOG.isDebugEnabled()) {
>         LOG.debug("Assigning container failed on node '" + node.getNodeName()
>             + " because queue resource usage is larger than MaxShare: "
>             + dumpState());
>       }
>       return false;
>     } else {
>       return true;
>     }
>   }
> {code}
> {code:java}
>   @Override
>   public Resource getResourceUsage() {
>     Resource usage = Resources.createResource(0);
>     readLock.lock();
>     try {
>       for (FSQueue child : childQueues) {
>         Resources.addTo(usage, child.getResourceUsage());
>       }
>     } finally {
>       readLock.unlock();
>     }
>     return usage;
>   }
> {code}


--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

---------------------------------------------------------------------
To unsubscribe, e-mail: yarn-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: yarn-issues-help@hadoop.apache.org