hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Sandy Ryza (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (YARN-3635) Get-queue-mapping should be a common interface of YarnScheduler
Date Thu, 16 Jul 2015 03:43:05 GMT

    [ https://issues.apache.org/jira/browse/YARN-3635?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14629139#comment-14629139

Sandy Ryza commented on YARN-3635:

[~leftnoteasy], apologies for this quick drive-by review - I am currently traveling.

The JIRA appears to be lacking a design-doc and I wasn't able to find documentation in the
patch.  The patch should ultimately include some detailed documentation, but I don't want
to ask this of you before OKing the approach.  In light of this, a few questions:
* What steps are required for the Fair Scheduler to integrate with this?
* Is a common way of configuration proposed?
* How does this differ from the current Fair Scheduler model?  To summarize:
** The FS model consists of a sequence of placement rules that the app is passed through.
** Each placement rule gets the chance to assign the app to a queue, reject the app, or pass.
 If it passes, the next rule gets a chance.
** A placement rule can base its decision on:
*** The submitting user.
*** The set of groups the submitting user belongs to.
*** The queue requested in the app submission.
*** A set of configuration options that are specific to the rule.
*** The set of queues given in the Fair Scheduler configuration.
** Rules are marked as "terminal" if they will never pass.  This helps to avoid misconfigurations
where users place rules after terminal rules.
** Rules have a "create" attribute which determines whether they can create a new queue or
whether they must assign to existing queues.
** Currently the set of placement rules is limited to what's implemented in YARN.  I.e. there's
no public pluggable rule support.
I noticed from Vinod's comment that this patch follows a similar structure.  Are there places
where my summary would not describe what's going on in this patch?

> Get-queue-mapping should be a common interface of YarnScheduler
> ---------------------------------------------------------------
>                 Key: YARN-3635
>                 URL: https://issues.apache.org/jira/browse/YARN-3635
>             Project: Hadoop YARN
>          Issue Type: Sub-task
>          Components: scheduler
>            Reporter: Wangda Tan
>            Assignee: Tan, Wangda
>         Attachments: YARN-3635.1.patch, YARN-3635.2.patch, YARN-3635.3.patch, YARN-3635.4.patch,
YARN-3635.5.patch, YARN-3635.6.patch
> Currently, both of fair/capacity scheduler support queue mapping, which makes scheduler
can change queue of an application after submitted to scheduler.
> One issue of doing this in specific scheduler is: If the queue after mapping has different
maximum_allocation/default-node-label-expression of the original queue, {{validateAndCreateResourceRequest}}
in RMAppManager checks the wrong queue.
> I propose to make the queue mapping as a common interface of scheduler, and RMAppManager
set the queue after mapping before doing validations.

This message was sent by Atlassian JIRA

View raw message