hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Arun Suresh (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (YARN-1503) Continuous resource-localization for YARN containers
Date Wed, 24 Aug 2016 12:39:20 GMT

    [ https://issues.apache.org/jira/browse/YARN-1503?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15434851#comment-15434851

Arun Suresh commented on YARN-1503:

Thanks for driving this [~jianhe]

I was wondering how this would tie into the NM Container state machine. Currently, a Container
is in the LOCALIZING state while resources are being localized and then it starts. What state
would the container be in now if re-Localization is allowed asynchronously ? What happens
if localization failed for that specific resource ? Should the Container continue running

Also looks like this overlaps a bit with what was being attempted in YARN-4876. In that JIRA,
we do a couple of things:
# it explicitly exposes an INIT state where the Container is LOCALIZED but not started yet.

# it allows running containers to be re-initialized (and re-started) with new resources.

The current prototype stops the container process, re-initializes and the re-starts it. It
would be pretty straight forward to have a policy to not stop the container (keep the process
running). Since we have explicit hooks in the container life-cycle, it would be possible to
version the resources and provide support for roll-back etc.

> Continuous resource-localization for YARN containers
> ----------------------------------------------------
>                 Key: YARN-1503
>                 URL: https://issues.apache.org/jira/browse/YARN-1503
>             Project: Hadoop YARN
>          Issue Type: Improvement
>            Reporter: Siddharth Seth
>            Assignee: Jian He
>         Attachments: Continuous-resource-localization.pdf
> We have a use case, where additional resources (jars, libraries etc) need to be made
available to an already running container. Ideally, we'd like this to be done via YARN (instead
of having potentially multiple containers per node download resources on their own).
> The goal is to enable NodeManagers to localize resources while container is running.
Today, resource-localization is always the first step before starting a container. It will
be useful if YARN can localize the resources continuously even while container is running.

This message was sent by Atlassian JIRA

To unsubscribe, e-mail: yarn-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: yarn-issues-help@hadoop.apache.org

View raw message