hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Patrick Liu (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (YARN-2257) Add user to queue mappings to automatically place users' apps into specific queues
Date Thu, 10 Jul 2014 10:18:06 GMT

    [ https://issues.apache.org/jira/browse/YARN-2257?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14057339#comment-14057339
] 

Patrick Liu commented on YARN-2257:
-----------------------------------

Hi, Vinod,
I think we could inject the user->queue mapping judgement in 'RMAppManager's method 'protected
synchronized void submitApplication':
      // Sanity checks
      if (submissionContext.getQueue() == null) {
        submissionContext.setQueue(YarnConfiguration.DEFAULT_QUEUE_NAME);
      }
      if (submissionContext.getApplicationName() == null) {
        submissionContext.setApplicationName(
            YarnConfiguration.DEFAULT_APPLICATION_NAME);
      }
All applications submitted to yarn will be launched by 'RMAppManager'.
'RMAppManager' will do sanity check, create a 'RMAppImpl' instance, and finally send the 'new
RMAppEvent(applicationId, RMAppEventType.START)' event.
When 'RMAppImpl' received the Event, it will change the state machine and do the transition.
The transition will launch the 'RMAppAttemptImpl', and start the 'RMAppAttemptImpl'.
Then the app will be scheduled by the specific scheduler. 
The only thing we need to injuect is the QUEUE in the submissionContext.
Like this:
// Precondition: set "user-as-default-queue" to false in yarn-site.xml
if(QueuePlacementRule.hasMappingForUser(user)) {
    submissionContext.setQueue(QueuePlacementRule.getQueue(user));
} else {
    submissionContext.setQueue(Default);
}

> Add user to queue mappings to automatically place users' apps into specific queues
> ----------------------------------------------------------------------------------
>
>                 Key: YARN-2257
>                 URL: https://issues.apache.org/jira/browse/YARN-2257
>             Project: Hadoop YARN
>          Issue Type: Improvement
>          Components: scheduler
>            Reporter: Patrick Liu
>            Assignee: Vinod Kumar Vavilapalli
>              Labels: features
>
> Currently, the fair-scheduler supports two modes, default queue or individual queue for
each user.
> Apparently, the default queue is not a good option, because the resources cannot be managed
for each user or group.
> However, individual queue for each user is not good enough. Especially when connecting
yarn with hive. There will be increasing hive users in a corporate environment. If we create
a queue for a user, the resource management will be hard to maintain.
> I think the problem can be solved like this:
> 1. Define user->queue mapping in Fair-Scheduler.xml. Inside each queue, use aclSubmitApps
to control user's ability.
> 2. Each time a user submit an app to yarn, if the user has mapped to a queue, the app
will be scheduled to that queue; otherwise, the app will be submitted to default queue.
> 3. If the user cannot pass aclSubmitApps limits, the app will not be accepted.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

Mime
View raw message