hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Carlo Curino (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (YARN-45) Scheduler feedback to AM to release containers
Date Thu, 11 Apr 2013 13:45:17 GMT

    [ https://issues.apache.org/jira/browse/YARN-45?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13628930#comment-13628930
] 

Carlo Curino commented on YARN-45:
----------------------------------

Sorry I read only your last comment and answered to that... 

Regarding your previous "larger" comment:
- what you propose is somewhat of a combination of 1 and 2 above, where we give the AM a hint
about what would happen at the container level if the pressure remains. I don't have strong
feelings about it, I agree it is easy to do, and maybe is a good compromised.
- however, I want to be able to maintain the tighter semantics of 1 (in case the ResourceRequest
is not specified in the message), which forces the AM to preempt exactly the set of containers
I am specifying. (now with very "targeted" ResourceRequest you can in practice 
do something similar). This covers use cases like the one I mentioned above.

We are posting more code in YARN-567 YARN-568 and YARN-569, check it out, it might provide
context for this conversation.  

                
> Scheduler feedback to AM to release containers
> ----------------------------------------------
>
>                 Key: YARN-45
>                 URL: https://issues.apache.org/jira/browse/YARN-45
>             Project: Hadoop YARN
>          Issue Type: Improvement
>          Components: resourcemanager
>            Reporter: Chris Douglas
>            Assignee: Carlo Curino
>         Attachments: YARN-45.patch
>
>
> The ResourceManager strikes a balance between cluster utilization and strict enforcement
of resource invariants in the cluster. Individual allocations of containers must be reclaimed-
or reserved- to restore the global invariants when cluster load shifts. In some cases, the
ApplicationMaster can respond to fluctuations in resource availability without losing the
work already completed by that task (MAPREDUCE-4584). Supplying it with this information would
be helpful for overall cluster utilization [1]. To this end, we want to establish a protocol
for the RM to ask the AM to release containers.
> [1] http://research.yahoo.com/files/yl-2012-003.pdf

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

Mime
View raw message