hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Sunil G (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (YARN-5889) Improve user-limit calculation in capacity scheduler
Date Fri, 16 Dec 2016 17:53:58 GMT

    [ https://issues.apache.org/jira/browse/YARN-5889?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15755070#comment-15755070
] 

Sunil G commented on YARN-5889:
-------------------------------

Generally I am also agreeing with the direction at which we are going towards.

Few points from end:
- For preemption calculation, one of the main problem could have been about the *free resources*
in the queue even when some users are over-utilizing its resource quota (these users could
become active/non-active). Because preemption module will be handling {free_resources + to_be_preempted_resources}
and need to think more like scheduler.
- Above point will play a big factor to decide when preemption need to kick in. It could be
when free/used become very smaller OR it could also be when there is a lot of violation from
few users which holds resource more than MULP but became non-active users.

As far as I understood, we will still have pre-computed user-limit model. But this cache will
be computed based on any event change on resource changes for non-active users. I think in
a busier and short-living app's cluster, we may recalculate more. But I think preemption module
will have a better accuracy.

On this note, could I update a patch with approach mentioned above. I think free resource
also need to be part to trigger preemption. But for user-limit calculation, I will be making
changes in {{ActiveUserManager}} to track of non-active-users as well with a state to reflect
changes in resource.

> Improve user-limit calculation in capacity scheduler
> ----------------------------------------------------
>
>                 Key: YARN-5889
>                 URL: https://issues.apache.org/jira/browse/YARN-5889
>             Project: Hadoop YARN
>          Issue Type: Bug
>          Components: capacity scheduler
>            Reporter: Sunil G
>            Assignee: Sunil G
>         Attachments: YARN-5889.v0.patch, YARN-5889.v1.patch, YARN-5889.v2.patch
>
>
> Currently user-limit is computed during every heartbeat allocation cycle with a write
lock. To improve performance, this tickets is focussing on moving user-limit calculation out
of heartbeat allocation flow.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: yarn-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: yarn-issues-help@hadoop.apache.org


Mime
View raw message