hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Wangda Tan (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (YARN-1963) Support priorities across applications within the same queue
Date Tue, 18 Nov 2014 19:47:34 GMT

    [ https://issues.apache.org/jira/browse/YARN-1963?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14216674#comment-14216674

Wangda Tan commented on YARN-1963:

Thanks [~sunilg], [~vinodkv] for your great effort on this!
I've just read through the design doc, some comments:

1) yarn.app.priority
How this get to be implemented? Does this means, any YARN application doesn't need change
a line of their code, can specify priority when submit the app using yarn CLI?
I think if this can be done, we should extend to other YARN parameters like queue, node-label-expression,

2) Specify only highest priority for queue and user
I found there are property like {{yarn.scheduler.root.<queue_name>.priority_label=high,low}}
and {{yarn.scheduler.capacity.root.<queue_name>.<priority_label>.acl=user1,user2}}.
I would perfer just specify only highest priority for queue and user. For example, it doesn't
make sense to me if priority = \{high,mid,low\}, and a queue can access \{high,low\} only.
Is there any benefit to specify individual priorities instead of highest priority?

3) User limit and priority
I think we shouldn't consider user limit within priority level, because the priority is not
a specific kind of resource. Comparing to node label, you cannot say, user-X of queue-A used
8G highest priority resource, but you can say, user-X of queue-A used 8G resource in node
with label=GPU. There's no difference for a 2G resource allocated to highest/lowest priority.

If we want to implement this,
bq. it will not be fair to schedule resources in a uniform manner for all application in a
queue with respect to user limits.
I suggest to add preemption within queue considering priority. Upon YARN-2069, we can considering
user-limit and priority together -- while enforcing user-limit, we always preempt from lower
priority applications.

Any thoughts? 


> Support priorities across applications within the same queue 
> -------------------------------------------------------------
>                 Key: YARN-1963
>                 URL: https://issues.apache.org/jira/browse/YARN-1963
>             Project: Hadoop YARN
>          Issue Type: New Feature
>          Components: api, resourcemanager
>            Reporter: Arun C Murthy
>            Assignee: Sunil G
>         Attachments: YARN Application Priorities Design.pdf
> It will be very useful to support priorities among applications within the same queue,
particularly in production scenarios. It allows for finer-grained controls without having
to force admins to create a multitude of queues, plus allows existing applications to continue
using existing queues which are usually part of institutional memory.

This message was sent by Atlassian JIRA

View raw message