hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Hitesh Shah (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (YARN-1609) Add Service Container type to NodeManager in YARN
Date Fri, 17 Jan 2014 23:50:20 GMT

    [ https://issues.apache.org/jira/browse/YARN-1609?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13875418#comment-13875418

Hitesh Shah commented on YARN-1609:

[~lvm] I think I understand what the goals are though I am still unclear on the specifics
as to what the real problems were when trying to launch processes directly via the NM as compared
to using the service container. 

  - Is the NM flow too slow for launching a new container or killing one? 
  - Is it difficult to monitoring of the status of a container?
  - Can the NM not handle launching/managing a large no. of containers on a single machine?

It would also be helpful to understand what kind of workarounds/hacks you needed to do so
as to understand the underlying problems that you faced. 


> Add Service Container type to NodeManager in YARN
> -------------------------------------------------
>                 Key: YARN-1609
>                 URL: https://issues.apache.org/jira/browse/YARN-1609
>             Project: Hadoop YARN
>          Issue Type: Improvement
>          Components: nodemanager
>    Affects Versions: 2.2.0
>            Reporter: Wangda Tan
>            Assignee: Wangda Tan
>         Attachments: Add Service Container type to NodeManager in YARN-V1.pdf
> From our work to support running OpenMPI on YARN (MAPREDUCE-2911), we found that it’s
important to have framework specific daemon process manage the tasks on each node directly.
The daemon process, most likely similar in other frameworks as well, provides critical services
to tasks running on that node(for example “wireup”, spawn user process in large numbers
at once etc). In YARN, it’s hard, if not possible, to have the those processes to be managed
by YARN. 
> We propose to extend the container model on NodeManager side to support “Service Container”
to run/manage such framework daemon/services process. We believe this is very useful to other
application framework developers as well.

This message was sent by Atlassian JIRA

View raw message