hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Bikas Saha (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (YARN-1040) De-link container life cycle from an Allocation
Date Fri, 25 Mar 2016 18:58:25 GMT

    [ https://issues.apache.org/jira/browse/YARN-1040?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15212248#comment-15212248

Bikas Saha commented on YARN-1040:

bq. Hmmm.. Given that launching multiple processes, being a new feature, I feel that it should
be fine to mandate the app to use new APIs, no ?
In Tez/Spark clearly using the ability to launch multiple processes in containers will need
the use of new APIs on the NM. And that could be an optional feature local to that part of
the code that can be safely added and then turned on/off in an isolated manner by users. That
is fine. But if to use the new API's for this one optional feature, we have to change Tez/Spark
to redo their AM-RM implementations and update all their internals regarding the concept of
allocations and containers (where what the entire code used to consider containers are now
allocations), then I hope we appreciate how destabilizing that change would be to those projects.

> De-link container life cycle from an Allocation
> -----------------------------------------------
>                 Key: YARN-1040
>                 URL: https://issues.apache.org/jira/browse/YARN-1040
>             Project: Hadoop YARN
>          Issue Type: Sub-task
>          Components: nodemanager
>    Affects Versions: 3.0.0
>            Reporter: Steve Loughran
>         Attachments: YARN-1040-rough-design.pdf
> The AM should be able to exec >1 process in a container, rather than have the NM automatically
release the container when the single process exits.
> This would let an AM restart a process on the same container repeatedly, which for HBase
would offer locality on a restarted region server.
> We may also want the ability to exec multiple processes in parallel, so that something
could be run in the container while a long-lived process was already running. This can be
useful in monitoring and reconfiguring the long-lived process, as well as shutting it down.

This message was sent by Atlassian JIRA

View raw message