hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Sandy Ryza (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (YARN-2257) Add user to queue mappings to automatically place users' apps into specific queues
Date Wed, 16 Jul 2014 17:22:05 GMT

    [ https://issues.apache.org/jira/browse/YARN-2257?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14063749#comment-14063749

Sandy Ryza commented on YARN-2257:

[~wangda] I agree with you that expecting admins to recompile Hadoop is unreasonable.  I don't
think we would expect admins to add rules.  The idea is more to have a small library of rules
we provide that fit into a common configuration framework.

If we were to add QueuePlacementRule that accepts a list of user->queue, and wanted to
express "accept the user's queue if they specify it in ApplicationSubmissionContext, otherwise
look for a user->queue mapping, if none is found, use the default queue", configuring it
according to the current Fair Scheduler format would look something like this:
  <rule name="specified" />
  <rule name="userToQueue">
    <mapping name="sally" queue="queue1">
    <mapping name="emilio" queue="queue2>
  <rule name="default"/>

> Add user to queue mappings to automatically place users' apps into specific queues
> ----------------------------------------------------------------------------------
>                 Key: YARN-2257
>                 URL: https://issues.apache.org/jira/browse/YARN-2257
>             Project: Hadoop YARN
>          Issue Type: Improvement
>          Components: scheduler
>            Reporter: Patrick Liu
>            Assignee: Vinod Kumar Vavilapalli
>              Labels: features
> Currently, the fair-scheduler supports two modes, default queue or individual queue for
each user.
> Apparently, the default queue is not a good option, because the resources cannot be managed
for each user or group.
> However, individual queue for each user is not good enough. Especially when connecting
yarn with hive. There will be increasing hive users in a corporate environment. If we create
a queue for a user, the resource management will be hard to maintain.
> I think the problem can be solved like this:
> 1. Define user->queue mapping in Fair-Scheduler.xml. Inside each queue, use aclSubmitApps
to control user's ability.
> 2. Each time a user submit an app to yarn, if the user has mapped to a queue, the app
will be scheduled to that queue; otherwise, the app will be submitted to default queue.
> 3. If the user cannot pass aclSubmitApps limits, the app will not be accepted.

This message was sent by Atlassian JIRA

View raw message