hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Chen He (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (YARN-1198) Capacity Scheduler headroom calculation does not work as expected
Date Mon, 21 Jul 2014 15:00:51 GMT

    [ https://issues.apache.org/jira/browse/YARN-1198?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14068594#comment-14068594

Chen He commented on YARN-1198:

With preemption, resource beyond guaranteed resource will be likely preempted. It should be
consider as a temporary resource.
One thing needs to be clarified about preemption. I think we can resolve YARN-2008 without
introducing preemption. Because if we allow preemption before define priority, it is wasting
time and resource to let thousand of  AMs to compete those "temporary resources" repeatedly.

priority is the most important factor in preemption of scheduling. I think, in this JIRA,
we are talking about how to efficiently and relatively accurate get headroom in capacity scheduler.
Preemption is another story. Here is how preemption defined in scheduling: 

"In computing, preemption is the act of temporarily interrupting a task being carried out
by a computer system, without requiring its cooperation, and with the intention of resuming
the task at a later time. Such a change is known as a context switch. It is normally carried
out by a privileged task or part of the system known as a preemptive scheduler, which has
the power to preempt, or interrupt, and later resume, other tasks in the system." refer to
Preemption from wikipedia [http://en.wikipedia.org/wiki/Preemption_%28computing%29]

> Capacity Scheduler headroom calculation does not work as expected
> -----------------------------------------------------------------
>                 Key: YARN-1198
>                 URL: https://issues.apache.org/jira/browse/YARN-1198
>             Project: Hadoop YARN
>          Issue Type: Bug
>            Reporter: Omkar Vinit Joshi
>            Assignee: Omkar Vinit Joshi
>         Attachments: YARN-1198.1.patch
> Today headroom calculation (for the app) takes place only when
> * New node is added/removed from the cluster
> * New container is getting assigned to the application.
> However there are potentially lot of situations which are not considered for this calculation
> * If a container finishes then headroom for that application will change and should be
notified to the AM accordingly.
> * If a single user has submitted multiple applications (app1 and app2) to the same queue
> ** If app1's container finishes then not only app1's but also app2's AM should be notified
about the change in headroom.
> ** Similarly if a container is assigned to any applications app1/app2 then both AM should
be notified about their headroom.
> ** To simplify the whole communication process it is ideal to keep headroom per User
per LeafQueue so that everyone gets the same picture (apps belonging to same user and submitted
in same queue).
> * If a new user submits an application to the queue then all applications submitted by
all users in that queue should be notified of the headroom change.
> * Also today headroom is an absolute number ( I think it should be normalized but then
this is going to be not backward compatible..)
> * Also  when admin user refreshes queue headroom has to be updated.
> These all are the potential bugs in headroom calculations

This message was sent by Atlassian JIRA

View raw message