hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Nathan Roberts (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (YARN-6775) CapacityScheduler: Improvements to assignContainers, avoid unnecessary canAssignToUser/Queue calls
Date Mon, 17 Jul 2017 14:56:00 GMT

    [ https://issues.apache.org/jira/browse/YARN-6775?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16089936#comment-16089936
] 

Nathan Roberts commented on YARN-6775:
--------------------------------------

Attached screenshots that show a couple of before/after metrics. Change went active early
on the 14th.
1) rmeventprocbusy is avg cpu busy of the Event Processor thread
2) rpceventprocessingtimeschedulerport is avg rpc processing time for the scheduler port.


> CapacityScheduler: Improvements to assignContainers, avoid unnecessary canAssignToUser/Queue
calls
> --------------------------------------------------------------------------------------------------
>
>                 Key: YARN-6775
>                 URL: https://issues.apache.org/jira/browse/YARN-6775
>             Project: Hadoop YARN
>          Issue Type: Sub-task
>          Components: capacityscheduler
>    Affects Versions: 2.8.1, 3.0.0-alpha3
>            Reporter: Nathan Roberts
>            Assignee: Nathan Roberts
>             Fix For: 3.0.0-beta1
>
>         Attachments: rmeventprocbusy.png, rpcprocessingtimeschedulerport.png, YARN-6775.001.patch,
YARN-6775.002.patch, YARN-6775.branch-2.002.patch, YARN-6775.branch-2.8.002.patch
>
>
> There are several things in assignContainers() that are done multiple times even though
the result cannot change (canAssignToUser, canAssignToQueue). Add some local caching to take
advantage of this fact.
> Will post patch shortly. Patch includes a simple throughput test that demonstrates when
we have users at their user-limit, the number of NodeUpdateSchedulerEvents we can process
can be improved from 13K/sec to 50K/sec.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

---------------------------------------------------------------------
To unsubscribe, e-mail: yarn-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: yarn-issues-help@hadoop.apache.org


Mime
View raw message