hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Bikas Saha (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (YARN-1404) Add support for unmanaged containers
Date Thu, 14 Nov 2013 02:49:21 GMT

    [ https://issues.apache.org/jira/browse/YARN-1404?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13822124#comment-13822124

Bikas Saha commented on YARN-1404:

It looks like the uber point under discussion here is what are the scenarios that we are targeting
and can we figure out an overall design and a coherent set of changes in YARN that address
those scenarios (if we decide that those scenarios make sense for YARN).
I think I agree that it would really helpful if the scenarios were properly laid out and a
design proposal made so that other people in the community can understand what is the goal
and why some changes are being suggested and made. Then finalize the design and creates a
set of sub-tasks or related tasks that can be clearly related to that scenario. We have already
been following that for RM Restart, RM HA, RM Work preserving restart, long running services
etc. Without this kind of a coherent approach, it may appear that set of disparate changes
are being made to the framework in an ad-hoc basis. The sum total of these changes may end
up altering the framework in unexpected ways for the community.
In this light, it would be really helpful if we create an umbrella jira for the goals we want
to meet with this and other Llama related changes. And publish an overall design plan that
presents a coherent picture of the proposed picture. It would also help if we made the already
committed & in-flight jiras related to this effort as sub-tasks of the umbrella jira.
Does that sound like a reasonable plan going forward?

> Add support for unmanaged containers
> ------------------------------------
>                 Key: YARN-1404
>                 URL: https://issues.apache.org/jira/browse/YARN-1404
>             Project: Hadoop YARN
>          Issue Type: New Feature
>          Components: nodemanager
>    Affects Versions: 2.2.0
>            Reporter: Alejandro Abdelnur
>            Assignee: Alejandro Abdelnur
>         Attachments: YARN-1404.patch
> Currently a container allocation requires to start a container process with the corresponding
NodeManager's node.
> For applications that need to use the allocated resources out of band from Yarn this
means that a dummy container process must be started.
> Impala/Llama is an example of such application which is currently starting a 'sleep 10y'
(10 years) process as the container process. And the resource capabilities are used out of
by and the Impala process collocated in the node. The Impala process ensures the processing
associated to that resources do not exceed the capabilities of the container. Also, if the
container is lost/preempted/killed, Impala stops using the corresponding resources.
> In addition, in the case of Llama, the current requirement of having a container process,
gets complicates when hard resource enforcement (memory -ContainersMonitor- or cpu -via cgroups-)
is enabled because Impala/Llama request resources with CPU and memory independently of each
other. Some requests are CPU only and others are memory only. Unmanaged containers solve this
problem as there is no underlying process with zero CPU or zero memory.

This message was sent by Atlassian JIRA

View raw message