Mailing-List: contact yarn-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
Reply-To: yarn-issues@hadoop.apache.org
Date: Thu, 25 Feb 2016 20:20:18 +0000 (UTC)
From: "Vinod Kumar Vavilapalli (JIRA)" <jira@apache.org>
To: yarn-issues@hadoop.apache.org
Message-ID: <JIRA.12662436.1375896716000.148134.1456431618340@Atlassian.JIRA>
In-Reply-To: <JIRA.12662436.1375896716000@Atlassian.JIRA>
References: <JIRA.12662436.1375896716000@Atlassian.JIRA>
 <JIRA.12662436.1375896716831@arcas>
Subject: [jira] [Commented] (YARN-1040) De-link container life cycle from
 the process and add ability to execute multiple processes in the same
 long-lived container
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit


    [ https://issues.apache.org/jira/browse/YARN-1040?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15167794#comment-15167794 ] 

Vinod Kumar Vavilapalli commented on YARN-1040:
-----------------------------------------------

Still catching up on some of your discussion, but quick comments on a few things that I care

We should really call them instead as {{Allocation}} and {{Container}}. That's the nomenclature I used at YARN-4726. Till now, YARN has combined the notion of Allocation and Container, which is the main de-linking that we need to do here. Process has an OS level connotation, and doesn't work well with more things in the picture like process-trees / multiple-processes / Docker (YARN-2466).

Taking this further, here's how the overall picture can look like the following

*ResourceManager*
 - RM only does allocations in the scheduling path. ResourceManager does all scheduling based on AllocationRequests and tracks Allocations..
 - RM receives AllocationRequests and returns fulfilled Allocations (and AllocationTokens) to AMs.

*Applications*
 - AM can in turn use the Allocations (and AllocationTokens) to launch multiple Containers on the NM.
    -- Simple case: AM only launches containers one-after-another. It's up to the app to do this.
    -- General case: AM launches multiple containers at the same time. This is essentially container-groups - we should keep this option open.
 - AMs can specify *single-use* AllocationRequests, at which point RM can simply return Containers and Container-Tokens (today's code-path).
 - Each Container exits when the process-tree / linux-container exits.
 - Each Container has an Identifier.
    -- For single-use allocation-requests, RM generates ContainerIDs
    -- For multi-use allocation-requests, apps could optionally specify a container-name that is scoped under the allocation. NM always returns a (generated or app-specified) ContainerID based off the allocation-ID. Essentially, allocationID + containerID is unique

*NodeManagers*
 - NodeManager also understands incoming Allocations and ties them to Container groups: it deals with Allocation activation/deactivation and Container start/stop. but does
    -- the following *decoupled from both allocations and containers*: localizations / re-localizations. This means local-resources should now have more scopes: container, allocation, application etc.
    -- *per allocation*: enforcement of resource-limits
    -- all of the following *per container*: (a) process/OS-container activation / deactivation, (b) process/OS-container auto-restart (YARN-4725)  log-aggregation

> De-link container life cycle from the process and add ability to execute multiple processes in the same long-lived container
> ----------------------------------------------------------------------------------------------------------------------------
>
>                 Key: YARN-1040
>                 URL: https://issues.apache.org/jira/browse/YARN-1040
>             Project: Hadoop YARN
>          Issue Type: Sub-task
>          Components: nodemanager
>    Affects Versions: 3.0.0
>            Reporter: Steve Loughran
>
> The AM should be able to exec >1 process in a container, rather than have the NM automatically release the container when the single process exits.
> This would let an AM restart a process on the same container repeatedly, which for HBase would offer locality on a restarted region server.
> We may also want the ability to exec multiple processes in parallel, so that something could be run in the container while a long-lived process was already running. This can be useful in monitoring and reconfiguring the long-lived process, as well as shutting it down.


--
This message was sent by Atlassian JIRA
(v6.3.4#6332)