hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Jason Lowe (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (YARN-5889) Improve user-limit calculation in capacity scheduler
Date Wed, 14 Dec 2016 16:26:58 GMT

    [ https://issues.apache.org/jira/browse/YARN-5889?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15748766#comment-15748766

Jason Lowe commented on YARN-5889:

+1 for keeping the user limit behavior consistent with what it does today.  Resources will
still be offered to apps in priority order (typically FIFO unless overridden by the app's
priority), and any user under the minimum limit should still be able to receive that resource.
 As such I'm not sure I agree with the proposal for active user limits.  It appears that limit
can go below the configured minimum if a lot of users are all asking for the available resources.
 That would change the behavior and affect app priorities beyond what the user limit does
today.  For purposes of unblocking the user limit preemption work, I'd like to preserve the
current computation so adding the preemption feature doesn't affect the behavior of the scheduler
even when the feature isn't used.

Speaking of preemption, we will need to be careful on how the two limits are applied to prevent
oscillations.  If resources are available then the active user limit should be applied as
it is today.  However if resources are not available then the preemption limit should be applied
not only to the user being targeted for preemption but also to the user who is requesting
the resource to trigger the preemption.  If we mix the two limits then preemption can oscillate
between two users because the active limit can be larger than the preemption limit.

For calculating the preemption all-user-limit, I think the equation can be simplified to absolute
queue capacity * MULP.  I personally do not want preemptions to give users resources beyond
the minimum.  For example if the MULP is configured to 25% and there are two users in the
queue, A at 70% usage and B at 30%, I'd rather not lose work by shooting A's containers to
give B resources beyond the configured minimum limit.  Preemptions can be very costly to an
application, so I don't think this should be completely fair (that's the job of FairScheduler).
 We should only preempt work to get users up to the minimum limit and only if others are above
the minimum limit. Thoughts on this?

> Improve user-limit calculation in capacity scheduler
> ----------------------------------------------------
>                 Key: YARN-5889
>                 URL: https://issues.apache.org/jira/browse/YARN-5889
>             Project: Hadoop YARN
>          Issue Type: Bug
>          Components: capacity scheduler
>            Reporter: Sunil G
>            Assignee: Sunil G
>         Attachments: YARN-5889.v0.patch, YARN-5889.v1.patch, YARN-5889.v2.patch
> Currently user-limit is computed during every heartbeat allocation cycle with a write
lock. To improve performance, this tickets is focussing on moving user-limit calculation out
of heartbeat allocation flow.

This message was sent by Atlassian JIRA

To unsubscribe, e-mail: yarn-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: yarn-issues-help@hadoop.apache.org

View raw message