hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Carlo Curino (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (YARN-1051) YARN Admission Control/Planner: enhancing the resource allocation model with time.
Date Wed, 12 Feb 2014 02:14:23 GMT

    [ https://issues.apache.org/jira/browse/YARN-1051?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13898677#comment-13898677

Carlo Curino commented on YARN-1051:

Correct... You got the flow right.

A couple more notes for clarity: 

For flexibility we are decoupling the creation of  a reservation from what applications are
ran into it. I could imagine to have something like Oozie 
to submit a reservation request (expressing the skyline of resource it will need for a pipeline
of jobs), get back a session id (practically a queue name),
 and then submit the various jobs to it. Dynamically the Plan follower will ensure the queue
exists,and has whatever capacity the admission control
dedicated to it in every moment in time. 

We plan to handle sudden collapses in cluster capacity (rack gone bad), by reconsidering the
plan as a whole (for now simple greedy replanner, deciding
what sessions to kill/reposition).  This is to be able to express reservations in absolute
terms (100 containers),
instead of relative (10% of capacity)... this is particularly important for gang jobs like
Giraph that cannot deal with partial allocations well.  

> YARN Admission Control/Planner: enhancing the resource allocation model with time.
> ----------------------------------------------------------------------------------
>                 Key: YARN-1051
>                 URL: https://issues.apache.org/jira/browse/YARN-1051
>             Project: Hadoop YARN
>          Issue Type: Improvement
>          Components: capacityscheduler, resourcemanager, scheduler
>            Reporter: Carlo Curino
>            Assignee: Carlo Curino
>         Attachments: YARN-1051-design.pdf, curino_MSR-TR-2013-108.pdf
> In this umbrella JIRA we propose to extend the YARN RM to handle time explicitly, allowing
users to "reserve" capacity over time. This is an important step towards SLAs, long-running
services, workflows, and helps for gang scheduling.

This message was sent by Atlassian JIRA

View raw message