hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Bikas Saha (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (YARN-45) Scheduler feedback to AM to release containers
Date Mon, 29 Apr 2013 07:48:23 GMT

    [ https://issues.apache.org/jira/browse/YARN-45?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13644327#comment-13644327
] 

Bikas Saha commented on YARN-45:
--------------------------------

My understanding is the the containers being presented in PreemptionMessage are going to be
preempted by the RM some time in the near future if the RM cannot find free resources elsewhere.
The AM's are not supposed to preempt the containers but they are encourage to checkpoint and
save work. The RM can always choose to not preempt these containers and so it would be sub-optimal
for the AM to kill these containers.
If we want to add additional information besides the set of containers-to-be-preempted then
I would prefer ResourceRequest (like it was in the original patch) and not Resource. Not only
is that symmetric but also allows the RM to provide additional information about where to
free containers. A smarter RM could potentially ask for resources to be preempted where the
under-allocated job wants it and a smart AM could help out by choosing containers close to
the desired locations. Secondly, Resource is too amorphous by itself. Asking an AM to free
50GB  does not tell it whether the RM needs 10*5 or 50*1. Without that information the AM
can end up freeing containers in a manner that does not help the RM to meet the request of
the under-allocated job, thus failing to meet quota and wasting work at the same time.
                
> Scheduler feedback to AM to release containers
> ----------------------------------------------
>
>                 Key: YARN-45
>                 URL: https://issues.apache.org/jira/browse/YARN-45
>             Project: Hadoop YARN
>          Issue Type: Sub-task
>          Components: resourcemanager
>            Reporter: Chris Douglas
>            Assignee: Carlo Curino
>         Attachments: YARN-45.patch, YARN-45.patch, YARN-45.patch, YARN-45.patch, YARN-45.patch,
YARN-45_summary_of_alternatives.pdf
>
>
> The ResourceManager strikes a balance between cluster utilization and strict enforcement
of resource invariants in the cluster. Individual allocations of containers must be reclaimed-
or reserved- to restore the global invariants when cluster load shifts. In some cases, the
ApplicationMaster can respond to fluctuations in resource availability without losing the
work already completed by that task (MAPREDUCE-4584). Supplying it with this information would
be helpful for overall cluster utilization [1]. To this end, we want to establish a protocol
for the RM to ask the AM to release containers.
> [1] http://research.yahoo.com/files/yl-2012-003.pdf

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

Mime
View raw message