hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Carlo Curino (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (YARN-2592) Preemption can kill containers to fulfil need of already over-capacity queue.
Date Wed, 24 Sep 2014 18:58:34 GMT

    [ https://issues.apache.org/jira/browse/YARN-2592?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14146728#comment-14146728
] 

Carlo Curino commented on YARN-2592:
------------------------------------

Preemption is trying to enforce the scheduler invariants. One of which is how over-capacity
is distributed among queues (weighted fairly on rightful capacity).  

I understand the desire to "protect" individual containers, and there will be many specific
examples we can come up with in which killing a container is a pity as it loses some work
(unless it handles the preemption message correctly and checkpoint its state), but long term
I think enforcing the invariants is more important (fair and predictable for users). The opposite
argument one can make is "why is queue B allowed to retain more over capacity than A?" if
this happens systematically or for long period of time is unnerving for users as much as some
lost work. 

Also note that preemption already has few built-in mechanisms (deadzones, and grace-periods)
designed to limit the impact on running tasks, are we sure that proper tuning of capacity/max-capcity/dead-zones/grace-periods
is not enough to remove 99% of the problem? This would be only an issue for long-running tasks
(exceeding 2x the grace periods), when run above the capacity + dead-zone of a queue but within
max-capacity. And only trigger, for a queue that is more over capacity than any other peer
queue, when the peer queue also has over-capacity needs exceeding free space, AND no under-capacity
queue is demanding the same resources.... we should make sure this is significant enough of
a scenario in practice to justify complexity of new configurables.

I am definitely opposed to make this the default behavior, but I agree with Jason that we
could add config parameters that allow to prevent preemption for over-capacity balancing.
I feel though this is a slippery slope, which I think might lead to many loopholes (protecting
AM being another one), that eventually will make configuring preemption and understanding
what is happening for the users very hard. 

I think promoting proper handling of preemption on the app side (i.e., checkpoint your state,
or redistributed your computation) is overall a healthier direction. 

My 2 cents..


> Preemption can kill containers to fulfil need of already over-capacity queue.
> -----------------------------------------------------------------------------
>
>                 Key: YARN-2592
>                 URL: https://issues.apache.org/jira/browse/YARN-2592
>             Project: Hadoop YARN
>          Issue Type: Bug
>    Affects Versions: 3.0.0, 2.5.1
>            Reporter: Eric Payne
>
> There are scenarios in which one over-capacity queue can cause preemption of another
over-capacity queue. However, since killing containers may lose work, it doesn't make sense
to me to kill containers to feed an already over-capacity queue.
> Consider the following:
> {code}
> root has A,B,C, total capacity = 90
> A.guaranteed = 30, A.pending = 5, A.current = 40
> B.guaranteed = 30, B.pending = 0, B.current = 50
> C.guaranteed = 30, C.pending = 0, C.current = 0
> {code}
> In this case, the queue preemption monitor will kill 5 resources from queue B so that
queue A can pick them up, even though queue A is already over its capacity. This could lose
any work that those containers in B had already done.
> Is there a use case for this behavior? It seems to me that if a queue is already over
its capacity, it shouldn't destroy the work of other queues. If the over-capacity queue needs
more resources, that seems to be a problem that should be solved by increasing its guarantee.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message