hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Xianyin Xin (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (YARN-4090) Make Collections.sort() more efficient in FSParentQueue.java
Date Thu, 27 Aug 2015 14:31:45 GMT

     [ https://issues.apache.org/jira/browse/YARN-4090?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel

Xianyin Xin updated YARN-4090:
    Attachment: sampling2.jpg

I construct a queue hierarchy with 3 levels,
       child1                  child2                child3
child1.child1~10, child2.child1~15, child3.child1~15
the number of leaf queues is 40. A total of 1000 apps running randomly on the leaf queues.
The sampling results show that about 2/3 of the cpu times of FSParentQueue.assignContainers()
was spent on Collections.sort(). In Collections.sort(), about 40% was spent on SchedulerAppplicationAttempt.getCurrentConsumption()
and about 36% was spent on Resources.substract(). The former time consuming is because FSParentQueue.getResourceUsage()
will make recursion on it's children, while for the latter time consuming, the clone() in
substract() takes much cpu time.

> Make Collections.sort() more efficient in FSParentQueue.java
> ------------------------------------------------------------
>                 Key: YARN-4090
>                 URL: https://issues.apache.org/jira/browse/YARN-4090
>             Project: Hadoop YARN
>          Issue Type: Improvement
>          Components: fairscheduler
>            Reporter: Xianyin Xin
>         Attachments: sampling1.jpg, sampling2.jpg
> Collections.sort() consumes too much time in a scheduling round.

This message was sent by Atlassian JIRA

View raw message