hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Johan Gustavsson (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (YARN-4066) Large number of queues choke fair scheduler
Date Mon, 28 Sep 2015 04:16:04 GMT

     [ https://issues.apache.org/jira/browse/YARN-4066?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel

Johan Gustavsson updated YARN-4066:
    Attachment: YARN-4066-3.patch

The formatting issues have been taken care of, sorry about that.

> Large number of queues choke fair scheduler
> -------------------------------------------
>                 Key: YARN-4066
>                 URL: https://issues.apache.org/jira/browse/YARN-4066
>             Project: Hadoop YARN
>          Issue Type: Bug
>          Components: fairscheduler
>    Affects Versions: 2.7.1
>            Reporter: Johan Gustavsson
>            Assignee: Johan Gustavsson
>         Attachments: YARN-4066-2.patch, YARN-4066-3.patch, yarn-4066-1.patch
> Due to synchronization and all the loops performed during queue creation, setting a large
amount of queues (12000+) will completely choke the scheduler. To deal with this some optimization
to "QueueManager.updateAllocationConfiguration(AllocationConfiguration queueConf)" should
be done to reduce the amount of unnesecary loops. The attached patch have been tested to work
with atleast 96000 queues. 

This message was sent by Atlassian JIRA

View raw message