hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Vinod Kumar Vavilapalli (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (YARN-1040) De-link container life cycle from the process and add ability to execute multiple processes in the same long-lived container
Date Thu, 25 Feb 2016 20:20:18 GMT

    [ https://issues.apache.org/jira/browse/YARN-1040?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15167794#comment-15167794
] 

Vinod Kumar Vavilapalli commented on YARN-1040:
-----------------------------------------------

Still catching up on some of your discussion, but quick comments on a few things that I care

We should really call them instead as {{Allocation}} and {{Container}}. That's the nomenclature
I used at YARN-4726. Till now, YARN has combined the notion of Allocation and Container, which
is the main de-linking that we need to do here. Process has an OS level connotation, and doesn't
work well with more things in the picture like process-trees / multiple-processes / Docker
(YARN-2466).

Taking this further, here's how the overall picture can look like the following

*ResourceManager*
 - RM only does allocations in the scheduling path. ResourceManager does all scheduling based
on AllocationRequests and tracks Allocations..
 - RM receives AllocationRequests and returns fulfilled Allocations (and AllocationTokens)
to AMs.

*Applications*
 - AM can in turn use the Allocations (and AllocationTokens) to launch multiple Containers
on the NM.
    -- Simple case: AM only launches containers one-after-another. It's up to the app to do
this.
    -- General case: AM launches multiple containers at the same time. This is essentially
container-groups - we should keep this option open.
 - AMs can specify *single-use* AllocationRequests, at which point RM can simply return Containers
and Container-Tokens (today's code-path).
 - Each Container exits when the process-tree / linux-container exits.
 - Each Container has an Identifier.
    -- For single-use allocation-requests, RM generates ContainerIDs
    -- For multi-use allocation-requests, apps could optionally specify a container-name that
is scoped under the allocation. NM always returns a (generated or app-specified) ContainerID
based off the allocation-ID. Essentially, allocationID + containerID is unique

*NodeManagers*
 - NodeManager also understands incoming Allocations and ties them to Container groups: it
deals with Allocation activation/deactivation and Container start/stop. but does
    -- the following *decoupled from both allocations and containers*: localizations / re-localizations.
This means local-resources should now have more scopes: container, allocation, application
etc.
    -- *per allocation*: enforcement of resource-limits
    -- all of the following *per container*: (a) process/OS-container activation / deactivation,
(b) process/OS-container auto-restart (YARN-4725)  log-aggregation

> De-link container life cycle from the process and add ability to execute multiple processes
in the same long-lived container
> ----------------------------------------------------------------------------------------------------------------------------
>
>                 Key: YARN-1040
>                 URL: https://issues.apache.org/jira/browse/YARN-1040
>             Project: Hadoop YARN
>          Issue Type: Sub-task
>          Components: nodemanager
>    Affects Versions: 3.0.0
>            Reporter: Steve Loughran
>
> The AM should be able to exec >1 process in a container, rather than have the NM automatically
release the container when the single process exits.
> This would let an AM restart a process on the same container repeatedly, which for HBase
would offer locality on a restarted region server.
> We may also want the ability to exec multiple processes in parallel, so that something
could be run in the container while a long-lived process was already running. This can be
useful in monitoring and reconfiguring the long-lived process, as well as shutting it down.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message