hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Carlo Curino (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (YARN-45) Scheduler feedback to AM to release containers
Date Thu, 11 Apr 2013 03:13:15 GMT

    [ https://issues.apache.org/jira/browse/YARN-45?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13628625#comment-13628625
] 

Carlo Curino commented on YARN-45:
----------------------------------

Agreed.

As for #1, your previous comments made us indeed "simplify" #1 as follows:
We inform the AM that a Set<ContainerID> will be killed unless he preempts them (the
exact same containers). We dropped the "trading these containers for equivalent ones" as we
agreed with your comments that would be too funky. 
The rationale behind including this simple container-based preemption is twofold: 
 a) it matches very well with what the FairScheduler does today (we simply provide a cheaper
form of preemption w.r.t. the straight-up kill it used to do), and 
 b) it allows for compact bookkeeping for "kill if no preemption happens" in a policy we wrote
to add preemption to the CapacityScheduler which seems to behave well.

As for #2 I totally agree this is important to have, and it has lots of potential since it
empowers the AM to make smart local decisions (it is well aligned with the overall spirit
of Yarn I think). 
We will handle this both in the RM and AM in future patches. Where "future" = we have the
code, but need a polish before posting.

Cheers,
Carlo
                
> Scheduler feedback to AM to release containers
> ----------------------------------------------
>
>                 Key: YARN-45
>                 URL: https://issues.apache.org/jira/browse/YARN-45
>             Project: Hadoop YARN
>          Issue Type: Improvement
>          Components: resourcemanager
>            Reporter: Chris Douglas
>            Assignee: Carlo Curino
>         Attachments: YARN-45.patch
>
>
> The ResourceManager strikes a balance between cluster utilization and strict enforcement
of resource invariants in the cluster. Individual allocations of containers must be reclaimed-
or reserved- to restore the global invariants when cluster load shifts. In some cases, the
ApplicationMaster can respond to fluctuations in resource availability without losing the
work already completed by that task (MAPREDUCE-4584). Supplying it with this information would
be helpful for overall cluster utilization [1]. To this end, we want to establish a protocol
for the RM to ask the AM to release containers.
> [1] http://research.yahoo.com/files/yl-2012-003.pdf

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

Mime
View raw message