hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Gergo Repas (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (YARN-8191) Fair scheduler: queue deletion without RM restart
Date Wed, 02 May 2018 16:10:00 GMT

    [ https://issues.apache.org/jira/browse/YARN-8191?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16461238#comment-16461238
] 

Gergo Repas commented on YARN-8191:
-----------------------------------

[~wilfreds] Thank you for the review! I uploaded an updated patch. My replies to your comments:
* > The queueMgr can never be null, it is created when the FS get created the null check
is not needed in the AllocationFileLoaderService
** I removed this check
* > You are adding a new method to the FS (hasApplicationAssignedToQueue) to check if an
application is submitted to the queue but the application attempt has not been created yet.
If we have a large number of queues and applications this count loop could become really expensive.
I would suggest that we track submitted applications in the FSLeafQueue. We can then easily
redefine empty as all 3 types (submit/run/non-run) being 0.
** Good point, I modified the code to according to your comment.
* > The root and root.default queue should not be marked dynamic when they get created
by the queue manager.
** They are actually marked as dynamic=false
* > I do not understand the change around the isEmpty / shutDownIfEmpty some code is commented
out and shutDownIfEmpty is never used
** I submitted this block comment/method by mistake, I removed it.
* > I think we should redefine isEmpty() in the QueueManager to take into account the submitted
apps instead of creating something new.
** Good point, I updated this method.
* > getDynamicQueueNames should rely on the isDynamic flag that is part of the queue and
not calculated.
** getDynamicQueueNames() is being used while we're setting the isDynamic flags, that's why
I could not rely on the flag itself

> Fair scheduler: queue deletion without RM restart
> -------------------------------------------------
>
>                 Key: YARN-8191
>                 URL: https://issues.apache.org/jira/browse/YARN-8191
>             Project: Hadoop YARN
>          Issue Type: Improvement
>          Components: fairscheduler
>    Affects Versions: 3.0.1
>            Reporter: Gergo Repas
>            Assignee: Gergo Repas
>            Priority: Major
>         Attachments: Queue Deletion in Fair Scheduler.pdf, YARN-8191.000.patch, YARN-8191.001.patch,
YARN-8191.002.patch, YARN-8191.003.patch, YARN-8191.004.patch
>
>
> The Fair Scheduler never cleans up queues even if they are deleted in the allocation
file, or were dynamically created and are never going to be used again. Queues always remain
in memory which leads to two following issues.
>  # Steady fairshares aren’t calculated correctly due to remaining queues
>  # WebUI shows deleted queues, which is confusing for users (YARN-4022).
> We want to support proper queue deletion without restarting the Resource Manager:
>  # Static queues without any entries that are removed from fair-scheduler.xml should
be deleted from memory.
>  # Dynamic queues without any entries should be deleted.
>  # RM Web UI should only show the queues defined in the scheduler at that point in time.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

---------------------------------------------------------------------
To unsubscribe, e-mail: yarn-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: yarn-issues-help@hadoop.apache.org


Mime
View raw message