hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Wangda Tan (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (YARN-6775) CapacityScheduler: Improvements to assignContainers()
Date Fri, 07 Jul 2017 23:47:00 GMT

    [ https://issues.apache.org/jira/browse/YARN-6775?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16078845#comment-16078845
] 

Wangda Tan commented on YARN-6775:
----------------------------------

Thanks [~nroberts], patch looks good to me in general, some minor comments regarding to changes
of LeafQueue

1) CachedUserLimit.canAssign is not necessary as we can set CachedUserLimit.reservation to
UNBOUNDED initially.
2) Directly set {{cul.reservation = rsrv}} could be problematic under async scheduling logic
since reserved resource of app could be updated while allocating.
3) Do you think is it necessary to add another Resource to track queue's verified_minimum_violated_reserved_resource
similar to user limit?

Few local var naming suggestions:
1) rsrv => appReserved
2) cul.reservation => minimumUnsatifiedReserved, does this look better?


> CapacityScheduler: Improvements to assignContainers()
> -----------------------------------------------------
>
>                 Key: YARN-6775
>                 URL: https://issues.apache.org/jira/browse/YARN-6775
>             Project: Hadoop YARN
>          Issue Type: Sub-task
>          Components: capacityscheduler
>    Affects Versions: 2.8.1, 3.0.0-alpha3
>            Reporter: Nathan Roberts
>            Assignee: Nathan Roberts
>         Attachments: YARN-6775.001.patch
>
>
> There are several things in assignContainers() that are done multiple times even though
the result cannot change (canAssignToUser, canAssignToQueue). Add some local caching to take
advantage of this fact.
> Will post patch shortly. Patch includes a simple throughput test that demonstrates when
we have users at their user-limit, the number of NodeUpdateSchedulerEvents we can process
can be improved from 13K/sec to 50K/sec.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

---------------------------------------------------------------------
To unsubscribe, e-mail: yarn-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: yarn-issues-help@hadoop.apache.org


Mime
View raw message