hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Yufei Gu (JIRA)" <j...@apache.org>
Subject [jira] [Resolved] (YARN-6947) The implementation of Schedulable#getResourceUsage so inefficiency that can reduce the performance of scheduling
Date Fri, 04 Aug 2017 08:10:00 GMT

     [ https://issues.apache.org/jira/browse/YARN-6947?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Yufei Gu resolved YARN-6947.
----------------------------
    Resolution: Duplicate

> The implementation of Schedulable#getResourceUsage so inefficiency that can reduce the
performance of scheduling
> ----------------------------------------------------------------------------------------------------------------
>
>                 Key: YARN-6947
>                 URL: https://issues.apache.org/jira/browse/YARN-6947
>             Project: Hadoop YARN
>          Issue Type: Improvement
>          Components: fairscheduler
>            Reporter: YunFan Zhou
>            Priority: Critical
>
> Each time the FairScheduler assign container, it will checks whether the resources used
by the queue exceed Max Share. However, our current calculation of the resources of the queue
is particularly inefficient, which recursively iterates over all child nodes, with high time
complexity. 
> We can refactor this logic by using lazy update way.
> {code:java}
>   @Override
>   public Resource assignContainer(FSSchedulerNode node) {
>     Resource assigned = Resources.none();
>     // If this queue is over its limit, reject
>     if (!assignContainerPreCheck(node)) {
>       return assigned;
>     }
> {code}
> {code:java}
>    * Helper method to check if the queue should attempt assigning resources
>    * 
>    * @return true if check passes (can assign) or false otherwise
>    */
>   boolean assignContainerPreCheck(FSSchedulerNode node) {
>     if (node.getReservedContainer() != null) {
>       if (LOG.isDebugEnabled()) {
>         LOG.debug("Assigning container failed on node '" + node.getNodeName()
>             + " because it has reserved containers.");
>       }
>       return false;
>     } else if (!Resources.fitsIn(getResourceUsage(), maxShare)) {
>       if (LOG.isDebugEnabled()) {
>         LOG.debug("Assigning container failed on node '" + node.getNodeName()
>             + " because queue resource usage is larger than MaxShare: "
>             + dumpState());
>       }
>       return false;
>     } else {
>       return true;
>     }
>   }
> {code}
> {code:java}
>   @Override
>   public Resource getResourceUsage() {
>     Resource usage = Resources.createResource(0);
>     readLock.lock();
>     try {
>       for (FSQueue child : childQueues) {
>         Resources.addTo(usage, child.getResourceUsage());
>       }
>     } finally {
>       readLock.unlock();
>     }
>     return usage;
>   }
> {code}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

---------------------------------------------------------------------
To unsubscribe, e-mail: yarn-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: yarn-issues-help@hadoop.apache.org


Mime
View raw message