hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Jason Lowe (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (YARN-3275) CapacityScheduler: Preemption happening on non-preemptable queues
Date Tue, 03 Mar 2015 16:32:06 GMT

    [ https://issues.apache.org/jira/browse/YARN-3275?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14345290#comment-14345290
] 

Jason Lowe commented on YARN-3275:
----------------------------------

Thanks for the patch, Eric!

bq. the expectation of our users is that if they are running a job on a non-preemptable queue,
their containers should never be preempted.

I completely agree with this.  IMHO the whole point of the preemption disable feature is to
guarantee a queue marked as such will never be preempted.  It's as if the entire preemption
feature was turned off from that queue's perspective.

Looking at the patch, I'm a bit worried about this part:

{code}
+      Resource absMaxCapIdealAssignedDelta = Resource.newInstance(0, 0);
+      if (Resources.greaterThanOrEqual(
+                        rc, clusterResource, maxCapacity, idealAssigned)) {
+        absMaxCapIdealAssignedDelta = Resources.subtract(maxCapacity, idealAssigned);
+      }
{code}

If the intent of this calculation is to guarantee none of the components of absMaxCapIdealAssignedDelta
are negative then I don't believe this accomplishes that goal.  I believe it's possible for
Resources.greaterThanOrEqual to return true yet subtracting the values will result in one
of the components to be negative.  For example, what if both resources are memory dominant,
maxCapacity has more memory than idealAssigned, but the opposite is true for vcores?  Subtracting
idealAssigned from maxCapacity will result in a positive memory component but a negative vcore
component.  If we need to make sure neither component goes negative then I think we need to
do a component-wise max with the zero resource rather than a comparision.

Also one style nit: we normally don't do one-liner conditionals without braces, so I'd like
to see the continue explicitly put in a block.  It might be useful to put a debug log statement
with the continue to note that we wanted to preempt this queue for some reason (and by how
much) but it was marked with preemption disabled.

> CapacityScheduler: Preemption happening on non-preemptable queues
> -----------------------------------------------------------------
>
>                 Key: YARN-3275
>                 URL: https://issues.apache.org/jira/browse/YARN-3275
>             Project: Hadoop YARN
>          Issue Type: Bug
>    Affects Versions: 2.7.0
>            Reporter: Eric Payne
>            Assignee: Eric Payne
>              Labels: capacity-scheduler
>         Attachments: YARN-3275.v1.txt
>
>
> YARN-2056 introduced the ability to turn preemption on and off at the queue level. In
cases where a queue goes over its absolute max capacity (YARN-3243, for example), containers
can be preempted from that queue, even though the queue is marked as non-preemptable.
> We are using this feature in large, busy clusters and seeing this behavior.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message