hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "YunFan Zhou (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (YARN-6361) FairScheduler: FSLeafQueue.fetchAppsWithDemand CPU usage is high with big queues
Date Fri, 18 Aug 2017 06:44:00 GMT

    [ https://issues.apache.org/jira/browse/YARN-6361?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16131803#comment-16131803
] 

YunFan Zhou commented on YARN-6361:
-----------------------------------

[~yufeigu] Thank you Yufei. 
It was in our production environment before, and I made a tradeoff in ensuring equitable sharing
and scheduling performance, sacrificing part of equitable sharing but providing scheduling
performance.
As you say, scheduling is a very generic component,  so we should ensure fairness as possible
as we can. 
So I later will try to come up with other sacrifice fairness and scheduling performance point.
And it is obvious to see that every time after scheduling, in fact, the order of the scheduling
entity actually does not change too much, which can be based on the optimization of the more
completely.

In addition, the patch I submitted last time is just optimizing the performance of the *fetchAppsWithDemand*,
please help me review the code? Thank you.

> FairScheduler: FSLeafQueue.fetchAppsWithDemand CPU usage is high with big queues
> --------------------------------------------------------------------------------
>
>                 Key: YARN-6361
>                 URL: https://issues.apache.org/jira/browse/YARN-6361
>             Project: Hadoop YARN
>          Issue Type: Sub-task
>            Reporter: Miklos Szegedi
>            Assignee: YunFan Zhou
>         Attachments: dispatcherthread.png, threads.png, YARN-6361.001.patch
>
>
> FSLeafQueue.fetchAppsWithDemand sorts the applications by the current policy. Most of
the time is spent in FairShareComparator.compare. We could improve this by doing the calculations
outside the sort loop {{(O\(n\))}} and we sorted by a fixed number inside instead {{O(n*log\(n\))}}.
This could be an performance issue when there are huge number of applications in a single
queue. The attachments shows the performance impact when there are 10k applications in one
queue.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

---------------------------------------------------------------------
To unsubscribe, e-mail: yarn-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: yarn-issues-help@hadoop.apache.org


Mime
View raw message