hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Arun C Murthy (JIRA)" <j...@apache.org>
Subject [jira] [Comment Edited] (YARN-1051) YARN Admission Control/Planner: enhancing the resource allocation model with time.
Date Thu, 20 Mar 2014 18:51:12 GMT

    [ https://issues.apache.org/jira/browse/YARN-1051?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13942122#comment-13942122
] 

Arun C Murthy edited comment on YARN-1051 at 3/20/14 6:50 PM:
--------------------------------------------------------------

More color on why I prefer priorities for reservations rather than adding/removing queues...

In vast majority of deployments, queues are an organizational/economic concept (e.g. per-department
queues are very common) and are queues (hierarchy, names etc.) are quite stable and well recognized
to point of being part of the institutional memory.

If we rely on adding/removing queues to provide reservations, I'm concerned it will cause
some confusion among both admins and users. For e.g. a user/admin trying to debug his application
will be quite challenged to figure demand/supply of resources when he has to go back in time
to reconstruct a programmatically generated queue hierarchy, particularly after it's long
gone.

Priorities, OTOH, is quite a familiar concept to admins (think unix 'nice'); and more importantly
is a natural fit to the problem at hand i.e. temporally increase/decrease the priority of
the application based on it's reservation at a point in time.

Furthermore, as I said previously, priorities are an often requested feature - especially
by admins.


was (Author: acmurthy):
More color on why I prefer priorities for reservations rather than adding/removing queues...

In vast majority of deployments, queues are an organizational/economic concept (e.g. per-department
queues are very common) and are queues (hierarchy, names etc.) are quite stable and well recognized
and part of the institutional memory.

If we rely on adding/removing queues to provide reservations, I'm concerned it will cause
some confusion among both admins and users. For e.g. a user/admin trying to debug his application
will be quite challenged to figure demand/supply of resources when he has to go back in time
to reconstruct a programmatically generated queue hierarchy, particularly after it's long
gone.

Priorities, OTOH, is quite a familiar concept to admins (think unix 'nice'); and more importantly
is a natural fit to the problem at hand i.e. temporally increase/decrease the priority of
the application based on it's reservation at a point in time.

Furthermore, as I said previously, priorities are an often requested feature - especially
by admins.

> YARN Admission Control/Planner: enhancing the resource allocation model with time.
> ----------------------------------------------------------------------------------
>
>                 Key: YARN-1051
>                 URL: https://issues.apache.org/jira/browse/YARN-1051
>             Project: Hadoop YARN
>          Issue Type: Improvement
>          Components: capacityscheduler, resourcemanager, scheduler
>            Reporter: Carlo Curino
>            Assignee: Carlo Curino
>         Attachments: YARN-1051-design.pdf, curino_MSR-TR-2013-108.pdf, techreport.pdf
>
>
> In this umbrella JIRA we propose to extend the YARN RM to handle time explicitly, allowing
users to "reserve" capacity over time. This is an important step towards SLAs, long-running
services, workflows, and helps for gang scheduling.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

Mime
View raw message