hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Johan Gustavsson (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (YARN-4066) Large number of queues choke fair scheduler
Date Mon, 28 Sep 2015 07:00:13 GMT

    [ https://issues.apache.org/jira/browse/YARN-4066?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14910113#comment-14910113

Johan Gustavsson commented on YARN-4066:

Thanks for re-running this. Looking at the tests in place I see no need to introduce new tests
for this patch, also this patch doesn't break the tests in place.

> Large number of queues choke fair scheduler
> -------------------------------------------
>                 Key: YARN-4066
>                 URL: https://issues.apache.org/jira/browse/YARN-4066
>             Project: Hadoop YARN
>          Issue Type: Bug
>          Components: fairscheduler
>    Affects Versions: 2.7.1
>            Reporter: Johan Gustavsson
>            Assignee: Johan Gustavsson
>         Attachments: YARN-4066-2.patch, YARN-4066-3.patch, yarn-4066-1.patch
> Due to synchronization and all the loops performed during queue creation, setting a large
amount of queues (12000+) will completely choke the scheduler. To deal with this some optimization
to "QueueManager.updateAllocationConfiguration(AllocationConfiguration queueConf)" should
be done to reduce the amount of unnesecary loops. The attached patch have been tested to work
with atleast 96000 queues. 

This message was sent by Atlassian JIRA

View raw message