hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Mayank Bansal (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (YARN-2069) CS queue level preemption should respect user-limits
Date Fri, 25 Jul 2014 21:22:40 GMT

    [ https://issues.apache.org/jira/browse/YARN-2069?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14074940#comment-14074940

Mayank Bansal commented on YARN-2069:

HI [~wangda] ,

Thanks for the review.

Let me explain what this algo is doing .

Lets say you have queueA in your cluster with capacity 30% allocated to it.
Now Queue A is using 50% resources. Queue A has 5 users with 20% user limit.
That means with each user is using 10% of the capacity of the cluster.

Now Another queueB is there with allocated capacity 70%.
Used capacity of queue B is 50%. Now if another application gets submitted to Queue B which
needs 10% capacity.

Now 10% capacity has to be claimed back from queue A .
So restoobtain = 10%
Targated user limit will be = 8% (This is always calculated based on how much we need to calim
back from user)

So based on the current alogorithm , it will take out 2% resources from every user and will
leave behind the balance for each users.
This will also be true if all the users are not using same number of resources so based on
this algo it will take out more from the users 
which are using more to balance till targated user limit.

Other thing this algo also does is it preempt application which is submitted last that means
if user1 has 2 application, it will try to take the maximum containers from the last application
submitted leaving behind the AM container however user limit will be honoured with combined
all applications in the queue.

This algo does not remove AM continer if its not absolutely needed, it goes to get all the
tasks containers first and then  consider AM containers.to be preempted.


> CS queue level preemption should respect user-limits
> ----------------------------------------------------
>                 Key: YARN-2069
>                 URL: https://issues.apache.org/jira/browse/YARN-2069
>             Project: Hadoop YARN
>          Issue Type: Sub-task
>          Components: capacityscheduler
>            Reporter: Vinod Kumar Vavilapalli
>            Assignee: Mayank Bansal
>         Attachments: YARN-2069-trunk-1.patch, YARN-2069-trunk-2.patch, YARN-2069-trunk-3.patch,
YARN-2069-trunk-4.patch, YARN-2069-trunk-5.patch, YARN-2069-trunk-6.patch, YARN-2069-trunk-7.patch
> This is different from (even if related to, and likely share code with) YARN-2113.
> YARN-2113 focuses on making sure that even if queue has its guaranteed capacity, it's
individual users are treated in-line with their limits irrespective of when they join in.
> This JIRA is about respecting user-limits while preempting containers to balance queue

This message was sent by Atlassian JIRA

View raw message