hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Wangda Tan (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (YARN-1197) Add container merge support in YARN
Date Fri, 13 Sep 2013 08:56:53 GMT

    [ https://issues.apache.org/jira/browse/YARN-1197?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13766343#comment-13766343

Wangda Tan commented on YARN-1197:

I don't know is it possible to add this in RM or NM side. 
And I think it should be easier to move some existing applications (OpenMPI, PBS, etc.) to
YARN platform, because such application should have their own daemons in old implementation,
and container merge can be helpful to leverage their original logic with less modifications
to be a resident of YARN :)
Welcome your suggestions and comments!
> Add container merge support in YARN
> -----------------------------------
>                 Key: YARN-1197
>                 URL: https://issues.apache.org/jira/browse/YARN-1197
>             Project: Hadoop YARN
>          Issue Type: Task
>          Components: api, nodemanager, resourcemanager
>    Affects Versions: 2.1.0-beta
>            Reporter: Wangda Tan
> Currently, YARN cannot support merge several containers in one node to a big container,
which can make us incrementally ask resources, merge them to a bigger one, and launch our
processes. The user scenario is,
> In some applications (like OpenMPI) has their own daemons in each node (one for each
node) in their original implementation, and their user's processes are directly launched by
its local daemon (like task-tracker in MRv1, but it's per-application). Many functionalities
are depended on the pipes created when a process forked by its father, like IO-forwarding,
process monitoring (it will do more logic than what NM did for us) and may cause some scalability
> A very common resource request in MPI world is, "give me 100G memory in the cluster,
I will launch 100 processes in this resource". In current YARN, we have following two choices
to make this happen,
> 1) Send allocation request with 1G memory iteratively, until we got 100G memories in
total. Then ask NM launch such 100 MPI processes. That will cause some problems like cannot
support IO-forwarding, processes monitoring, etc. as mentioned above.
> 2) Send a larger resource request, like 10G. But we may encounter following problems,
>    2.1 Such a large resource request is hard to get at one time.
>    2.2 We cannot use other resources more than the number we specified in the node (we
can only launch one daemon in one node).
>    2.3 Hard to decide how much resource to ask.
> So my proposal is,
> 1) We can incrementally send resource request with small resources like before, until
we get enough resources in total
> 2) Merge resource in the same node, make only one big container in each node
> 3) Launch daemons in each node, and the daemon will spawn its local processes and manage
> For example,
> We need to run 10 processes, 1G for each, finally we got
> container 1, 2, 3, 4, 5 in node1.
> container 6, 7, 8 in node2.
> container 9, 10 in node3.
> Then we will,
> merge [1, 2, 3, 4, 5] to container_11 with 5G, launch a daemon, and the daemon will launch
5 processes
> merge [6, 7, 8] to container_12 with 3G, launch a daemon, and the daemon will launch
3 processes
> merge [9, 10] to container_13 with 2G, launch a daemon, and the daemon will launch 2

This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

View raw message