hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Eric Payne (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (YARN-2592) Preemption can kill containers to fulfil need of already over-capacity queue.
Date Fri, 27 Feb 2015 15:10:05 GMT

    [ https://issues.apache.org/jira/browse/YARN-2592?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14340241#comment-14340241

Eric Payne commented on YARN-2592:

Closing this, since it is expected that as long as there are available resources, queue usage
should grow evenly based on percentage of absolute capacity, even when preemption can happen
to fill this growth as long as the absolute max capacity is not reached and the queues are
growing evenly.

> Preemption can kill containers to fulfil need of already over-capacity queue.
> -----------------------------------------------------------------------------
>                 Key: YARN-2592
>                 URL: https://issues.apache.org/jira/browse/YARN-2592
>             Project: Hadoop YARN
>          Issue Type: Bug
>    Affects Versions: 3.0.0, 2.5.1
>            Reporter: Eric Payne
> There are scenarios in which one over-capacity queue can cause preemption of another
over-capacity queue. However, since killing containers may lose work, it doesn't make sense
to me to kill containers to feed an already over-capacity queue.
> Consider the following:
> {code}
> root has A,B,C, total capacity = 90
> A.guaranteed = 30, A.pending = 5, A.current = 40
> B.guaranteed = 30, B.pending = 0, B.current = 50
> C.guaranteed = 30, C.pending = 0, C.current = 0
> {code}
> In this case, the queue preemption monitor will kill 5 resources from queue B so that
queue A can pick them up, even though queue A is already over its capacity. This could lose
any work that those containers in B had already done.
> Is there a use case for this behavior? It seems to me that if a queue is already over
its capacity, it shouldn't destroy the work of other queues. If the over-capacity queue needs
more resources, that seems to be a problem that should be solved by increasing its guarantee.

This message was sent by Atlassian JIRA

View raw message