hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Djellel Eddine Difallah (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (YARN-897) CapacityScheduler wrongly sorted queues
Date Wed, 03 Jul 2013 19:58:20 GMT

    [ https://issues.apache.org/jira/browse/YARN-897?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13699373#comment-13699373

Djellel Eddine Difallah commented on YARN-897:

We spotted this bug while experimenting on dynamic queues updates. The TreeSet methods .contains()
and .remove() failed on retrieving a queue that we knew was there, and that gave us a hint
that the tree was unsorted properly.
The attached test is a [simple junit test | https://issues.apache.org/jira/secure/attachment/12590676/TestBugParentQueue.java]
inspired by the already available capacity scheduler tests. It does simulate the scenario
that [~curino] describes above and displays the order in which the childQueues is left after
a couple of container assignments and completions.
I will post a first version of a patch that re-inserts the recently completed container's
queue (and all its parents) into their respective parents' childQueues. 
> CapacityScheduler wrongly sorted queues
> ---------------------------------------
>                 Key: YARN-897
>                 URL: https://issues.apache.org/jira/browse/YARN-897
>             Project: Hadoop YARN
>          Issue Type: Bug
>          Components: capacityscheduler
>            Reporter: Djellel Eddine Difallah
>         Attachments: TestBugParentQueue.java
> The childQueues of a ParentQueue are stored in a TreeSet where UsedCapacity defines the
sort order. This ensures the queue with least UsedCapacity to receive resources next. On containerAssignment
we correctly update the order, but we miss to do so on container completions. This corrupts
the TreeSet structure, and under-capacity queues might starve for resources.

This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

View raw message