hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Jason Lowe (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (YARN-2113) Add cross-user preemption within CapacityScheduler's leaf-queue
Date Tue, 18 Apr 2017 22:39:41 GMT

    [ https://issues.apache.org/jira/browse/YARN-2113?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15973638#comment-15973638

Jason Lowe commented on YARN-2113:

Is a deadzone the proper way to fix this?  I'm thinking of a case where the user has a particularly
large container, larger than the dead zone.  It will still flap in this case, correct?  Seems
like we should preempt not until we fall below the user limit but instead until the _next_
container we would preempt would put the user at or below their limit.  The scheduler essentially
entitles a user to one container beyond the user limit.  If we preempt down to a point at
or below the user's limit then we've gone one container too far, and the scheduler could very
well turn around and give the container right back.

Preempting down to one container before we meet or dip below the user limit has the advantage
that there's not yet another config to setup correctly.  However it brings up an interesting
scenario where killling the youngest container would lower the utilization below the user's
limit but killing older, smaller containers would not.

> Add cross-user preemption within CapacityScheduler's leaf-queue
> ---------------------------------------------------------------
>                 Key: YARN-2113
>                 URL: https://issues.apache.org/jira/browse/YARN-2113
>             Project: Hadoop YARN
>          Issue Type: Sub-task
>          Components: scheduler
>            Reporter: Vinod Kumar Vavilapalli
>            Assignee: Sunil G
>         Attachments: TestNoIntraQueuePreemptionIfBelowUserLimitAndDifferentPrioritiesWithExtraUsers.txt,
YARN-2113.0001.patch, YARN-2113.0002.patch, YARN-2113.0003.patch, YARN-2113.0004.patch, YARN-2113.0005.patch,
YARN-2113.0006.patch, YARN-2113.0007.patch, YARN-2113.v0.patch
> Preemption today only works across queues and moves around resources across queues per
demand and usage. We should also have user-level preemption within a queue, to balance capacity
across users in a predictable manner.

This message was sent by Atlassian JIRA

To unsubscribe, e-mail: yarn-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: yarn-issues-help@hadoop.apache.org

View raw message