hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Nathan Roberts (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (YARN-6775) CapacityScheduler: Improvements to assignContainers()
Date Tue, 11 Jul 2017 21:43:00 GMT

    [ https://issues.apache.org/jira/browse/YARN-6775?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16083045#comment-16083045

Nathan Roberts commented on YARN-6775:

Thanks [~leftnoteasy] for the review.

bq. 1) CachedUserLimit.canAssign is not necessary as we can set CachedUserLimit.reservation
to UNBOUNDED initially.
I think it is necessary because we need to keep track of the largest reservation for which
canAssignToUser() returns false. Anything smaller than the largest we've already calculated
we know won't work so we can avoid the call. Therefore we can't start at UNBOUNDED and work
our way down. 

bq. 2) Directly set cul.reservation = rsrv could be problematic under async scheduling logic
since reserved resource of app could be updated while allocating.
Is this something we need to address? Couldn't this be mutating between the various lookups
that are already occurring in today's assignContainers()?

bq. 3) Do you think is it necessary to add another Resource to track queue's verified_minimum_violated_reserved_resource
similar to user limit?
My thought was we'll quickly run across an app that has no reservation and be able to skip
assignToQueue() check from that point forward. The check against Resources.none() should be
very fast compared to a resource comparison. I'm open to keeping track of the minimum though
if you feel there would be sufficient gain.

Naming suggestions look good. I'll clean those up.(although minimum is actually a maximum)

> CapacityScheduler: Improvements to assignContainers()
> -----------------------------------------------------
>                 Key: YARN-6775
>                 URL: https://issues.apache.org/jira/browse/YARN-6775
>             Project: Hadoop YARN
>          Issue Type: Sub-task
>          Components: capacityscheduler
>    Affects Versions: 2.8.1, 3.0.0-alpha3
>            Reporter: Nathan Roberts
>            Assignee: Nathan Roberts
>         Attachments: YARN-6775.001.patch
> There are several things in assignContainers() that are done multiple times even though
the result cannot change (canAssignToUser, canAssignToQueue). Add some local caching to take
advantage of this fact.
> Will post patch shortly. Patch includes a simple throughput test that demonstrates when
we have users at their user-limit, the number of NodeUpdateSchedulerEvents we can process
can be improved from 13K/sec to 50K/sec.

This message was sent by Atlassian JIRA

To unsubscribe, e-mail: yarn-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: yarn-issues-help@hadoop.apache.org

View raw message