hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Eric Payne (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (YARN-2113) Add cross-user preemption within CapacityScheduler's leaf-queue
Date Fri, 17 Mar 2017 17:24:42 GMT

    [ https://issues.apache.org/jira/browse/YARN-2113?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15930339#comment-15930339
] 

Eric Payne commented on YARN-2113:
----------------------------------

Hi [~sunilg]. Thanks for this patch. I'm excited to look at it.

There does seem to be a problem with preempting when it shouldn't. If 2 apps are in a queue,
both are asking for resources, and one is over and one is under user limit, the app that is
under its user limit will get preempted.

Adding the he following unit test to {{TestProportionalCapacityPreemptionPolicyIntraQueueUserLimit}}
should demonstrate this. I have also observed this behavior in my manual testing.
{code}
  public void testIntraQueuePreemptionWithTwoRequstingUsers()
      throws IOException {

    // Set max preemption limit as 50%.
    conf.setFloat(CapacitySchedulerConfiguration.
        INTRAQUEUE_PREEMPTION_MAX_ALLOWABLE_LIMIT,
        (float) 0.5);

    String labelsConfig = "=100,true;";
    String nodesConfig = // n1 has no label
        "n1= res=100";
    String queuesConfig =
        // guaranteed,max,used,pending,reserved
        "root(=[100 100 100 20 0]);" + // root
            "-a(=[100 100 100 20 0])"; // b

    String appsConfig =
    // queueName\t(priority,resource,host,expression,#repeat,reserved,pending)
        "a\t" // app1 in a
            + "(1,1,n1,,100,false,10,user1);" + // app1 a
            "a\t" // app2 in a
            + "(1,1,n1,,40,false,10,user2)";

    buildEnv(labelsConfig, nodesConfig, queuesConfig, appsConfig);
    policy.editSchedule();

    // app2 needs more resource and its well under its user-limit. Hence preempt
    // resources from app1.
    verify(mDisp, times(0)).handle(argThat(
        new TestProportionalCapacityPreemptionPolicy.IsPreemptionRequestFor(
            getAppAttemptId(2))));
  }
{code}

> Add cross-user preemption within CapacityScheduler's leaf-queue
> ---------------------------------------------------------------
>
>                 Key: YARN-2113
>                 URL: https://issues.apache.org/jira/browse/YARN-2113
>             Project: Hadoop YARN
>          Issue Type: Sub-task
>          Components: scheduler
>            Reporter: Vinod Kumar Vavilapalli
>            Assignee: Sunil G
>         Attachments: YARN-2113.v0.patch
>
>
> Preemption today only works across queues and moves around resources across queues per
demand and usage. We should also have user-level preemption within a queue, to balance capacity
across users in a predictable manner.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

---------------------------------------------------------------------
To unsubscribe, e-mail: yarn-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: yarn-issues-help@hadoop.apache.org


Mime
View raw message