hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Alejandro Abdelnur (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (YARN-373) Allow an AM to reuse the resources allocated to container for a new container
Date Tue, 05 Feb 2013 00:14:13 GMT

    [ https://issues.apache.org/jira/browse/YARN-373?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13570782#comment-13570782
] 

Alejandro Abdelnur commented on YARN-373:
-----------------------------------------

Hitesh,

Didn't dive into the whole approach yet, first wanted to 'socialize' the idea. Now let me
answer with my current thoughts.

For this use case I was not thinking about resizing 'inflight' containers, while we could
resize easily on CPU, for memory would be quite difficult. 

The use case is about shortcutting getting resources for a container by reusing the same (or
less) resources being freed up by a terminating container in the same node. By doing this
you don't have to go to all the way to the scheduler and compete/wait for those resources
to become avail. In short, recycling resources the AM already got.

The terminating container would still exit, not changing the notion of completion of a container.
The container using the recycled resources would be a fresh new container process. (Otherwise
we could not shrink in memory).

Regarding localized resources, a new resource localization would be done.
                
> Allow an AM to reuse the resources allocated to container for a new container
> -----------------------------------------------------------------------------
>
>                 Key: YARN-373
>                 URL: https://issues.apache.org/jira/browse/YARN-373
>             Project: Hadoop YARN
>          Issue Type: Improvement
>          Components: resourcemanager
>    Affects Versions: 2.0.3-alpha
>            Reporter: Alejandro Abdelnur
>            Assignee: Alejandro Abdelnur
>
> When a container completes, instead the corresponding resources being freed up, it should
be possible for the AM to reuse the assigned resources for a new container.
> As part of the reallocation, the AM would notify the RM about partial resources being
freed up and the RM would make the necessary corrections in the corresponding node.
> With this functionality, an AM can ensure it gets a container in the same node where
previous containers run.
> This will allow getting rid of the ShuffleHandler as a service in the NMs and run it
as regular container task of the corresponding AM. In this case, the reallocation would reduce
the CPU/MEM obtained for the original container to the what is needed for serving the shuffle.
Note that in this example the MR AM would only do this reallocation for one of the many tasks
that may have run in a particular node (as a single shuffle task could serve all the map outputs
from all map tasks run in that node). 

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

Mime
View raw message