hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Carlo Curino (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (YARN-45) Scheduler feedback to AM to release containers
Date Mon, 29 Apr 2013 17:24:17 GMT

    [ https://issues.apache.org/jira/browse/YARN-45?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13644664#comment-13644664
] 

Carlo Curino commented on YARN-45:
----------------------------------

[~acmurthy] I see your point, which was in fact reflected more clearly in our initial proposal.
The only caveat is not to make this a capacity-only protocol (which you are not, but I wanted
to reiterate that there are other use cases).  

I like [~bikassaha] and [~chris.douglas] spin on it (i.e., using ResourceRequest), as it gives
us the immediate "capacity angle", but will eventually allow to evolve the implementations
towards something richer (e.g., the preempt on behalf of a specific request
that Bikas considered before) without impact to the protocols. 

I think there is a slightly cleaner version of Chris's proposal: 
use ResourceRequest and to represent a request that only cares about overall capacity we could
express the ResourceRequest as a multiple of the minimum allocation (i.e., if we want 100GB
of RAM back and min_container size is 1GB we ask for 100 x 1GB containers). This achieves
Chris's proposal with a slightly prettier use of ResourceRequest. Note that there are size-matching
issues (e.g., you have 1.5GB containers and I ask for 1x1GB containers, but we have very similar
problems with Resource).

I would say that as Chris pointed out [these semantics | https://issues.apache.org/jira/browse/YARN-45?focusedCommentId=13628950&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13628950]
plus the use of ResourceRequest I propose here as a minor variation on Chris's take should
cover Arun's and Bika's comments (and I believe also the prior 45+ messages). 

Thoughts?



                
> Scheduler feedback to AM to release containers
> ----------------------------------------------
>
>                 Key: YARN-45
>                 URL: https://issues.apache.org/jira/browse/YARN-45
>             Project: Hadoop YARN
>          Issue Type: Sub-task
>          Components: resourcemanager
>            Reporter: Chris Douglas
>            Assignee: Carlo Curino
>         Attachments: YARN-45.patch, YARN-45.patch, YARN-45.patch, YARN-45.patch, YARN-45.patch,
YARN-45_summary_of_alternatives.pdf
>
>
> The ResourceManager strikes a balance between cluster utilization and strict enforcement
of resource invariants in the cluster. Individual allocations of containers must be reclaimed-
or reserved- to restore the global invariants when cluster load shifts. In some cases, the
ApplicationMaster can respond to fluctuations in resource availability without losing the
work already completed by that task (MAPREDUCE-4584). Supplying it with this information would
be helpful for overall cluster utilization [1]. To this end, we want to establish a protocol
for the RM to ask the AM to release containers.
> [1] http://research.yahoo.com/files/yl-2012-003.pdf

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

Mime
View raw message