hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Carlo Curino (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (YARN-1051) YARN Admission Control/Planner: enhancing the resource allocation model with time.
Date Thu, 20 Mar 2014 20:09:46 GMT

    [ https://issues.apache.org/jira/browse/YARN-1051?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13942214#comment-13942214

Carlo Curino commented on YARN-1051:

Arun, I think the current design point is closer to what you describe than it seems (we changed
a fair bit from the early conversation we had). 

We created two new types of queues: InvetoryQueue and SessionQueue, which respectively inherit
form ParentQueue and LeafQueue... 
The distribution of resources among SessionQueues performed by the InventoryQueue (inherited
by ParentQueue, but without a requirement
of capacity of children to sum-up to 100%) corresponds very much to the priority mechanism
you refer to (a SessionQueue with high nominal
 capacity and low utilization is favored, etc.). So in principle, we could change InventoryQueue
to track apps directly with a priority list.

On the other hand, since we envision each SessionQueue to potentially being used to submit
multiple jobs (change of perspective from early
design, e.g., hive, pig multi-job queries, or pipelines), inheriting from the LeafQueue and
imposing the classic FIFO internal behavior + delay 
scheduling + other limits seems, allows to:
1)  make the notion of "Session" to be a rather consistent extension of a "Queue", a session
is a queue with some time-evolving properties (e.g., capacity).
2) reuse lots of tracking structures and well tested code.

Supporting this notion of session by assigning individual priorities to jobs that share a
session, and having multiple sessions per inventory seems harder to get right and maintain.

We would also be happy to meet and talk this through, and then report in JIRA the result of
our conversation.

> YARN Admission Control/Planner: enhancing the resource allocation model with time.
> ----------------------------------------------------------------------------------
>                 Key: YARN-1051
>                 URL: https://issues.apache.org/jira/browse/YARN-1051
>             Project: Hadoop YARN
>          Issue Type: Improvement
>          Components: capacityscheduler, resourcemanager, scheduler
>            Reporter: Carlo Curino
>            Assignee: Carlo Curino
>         Attachments: YARN-1051-design.pdf, curino_MSR-TR-2013-108.pdf, techreport.pdf
> In this umbrella JIRA we propose to extend the YARN RM to handle time explicitly, allowing
users to "reserve" capacity over time. This is an important step towards SLAs, long-running
services, workflows, and helps for gang scheduling.

This message was sent by Atlassian JIRA

View raw message