hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Gergo Repas (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (YARN-8191) Fair scheduler: queue deletion without RM restart
Date Wed, 09 May 2018 09:53:00 GMT

    [ https://issues.apache.org/jira/browse/YARN-8191?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16468634#comment-16468634
] 

Gergo Repas commented on YARN-8191:
-----------------------------------

Thanks [~wilfreds], I'll address those issues. Yes, I'm still working on the patch, once it's
good I'll reply to your comments.

For the record, the reason behind the reservation test failures is the fact that the
{code}
<?xml version="1.0"?>
<allocations>
<queue name="root">
  <aclAdministerApps>someuser </aclAdministerApps>
  <queue name="default">
    <aclAdministerApps>someuser </aclAdministerApps>
  </queue>
  <queue name="dedicated">
    <reservation>
    </reservation>
    <aclAdministerApps>someuser </aclAdministerApps>
  </queue>
</queue>
<defaultQueueSchedulingPolicy>drf</defaultQueueSchedulingPolicy>
</allocations>
{code}
test allocation file is being parsed as: {{PARENT=[root, root.dedicated], LEAF=[root.default,
root.dedicated]}}

> Fair scheduler: queue deletion without RM restart
> -------------------------------------------------
>
>                 Key: YARN-8191
>                 URL: https://issues.apache.org/jira/browse/YARN-8191
>             Project: Hadoop YARN
>          Issue Type: Improvement
>          Components: fairscheduler
>    Affects Versions: 3.0.1
>            Reporter: Gergo Repas
>            Assignee: Gergo Repas
>            Priority: Major
>         Attachments: Queue Deletion in Fair Scheduler.pdf, YARN-8191.000.patch, YARN-8191.001.patch,
YARN-8191.002.patch, YARN-8191.003.patch, YARN-8191.004.patch, YARN-8191.005.patch, YARN-8191.006.patch,
YARN-8191.007.patch
>
>
> The Fair Scheduler never cleans up queues even if they are deleted in the allocation
file, or were dynamically created and are never going to be used again. Queues always remain
in memory which leads to two following issues.
>  # Steady fairshares aren’t calculated correctly due to remaining queues
>  # WebUI shows deleted queues, which is confusing for users (YARN-4022).
> We want to support proper queue deletion without restarting the Resource Manager:
>  # Static queues without any entries that are removed from fair-scheduler.xml should
be deleted from memory.
>  # Dynamic queues without any entries should be deleted.
>  # RM Web UI should only show the queues defined in the scheduler at that point in time.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

---------------------------------------------------------------------
To unsubscribe, e-mail: yarn-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: yarn-issues-help@hadoop.apache.org


Mime
View raw message