hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Chris Douglas (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (YARN-5621) Support LinuxContainerExecutor to create symlinks for continuously localized resources
Date Tue, 20 Sep 2016 18:14:20 GMT

    [ https://issues.apache.org/jira/browse/YARN-5621?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15507320#comment-15507320

Chris Douglas commented on YARN-5621:

I think I see where the CL proposal was unclear.

It is an alternative to CE changes; container start remains as-is. The proposal was scoped
only to localizing resources for running containers. The CE is agnostic to new/running containers
for an application- it may be used by both, concurrently. By adding a new command {{LINK}}
to its protocol, the NM can instruct the {{ContainerLocalizer}} to create a symlink to a resource
for a running container. Again, these commands could be grouped.

> a case that already exists for containers on the same node requesting the same resource
Do you mean this is an existing implemented functionality or this is an existing use-case?

Neither. The case where running containers (c ~1x~, c ~2y~) for different applications (a
~1~, a ~2~) request the same resource _R_ exists. Both will start {{ContainerLocalizer}} instances,
but only one will download the resource to the private cache. In the CL proposal, this is
the same as rollback, where the CL starts, heartbeats, then receives a command to LINK an
existing resource without downloading anything. By "a case that already exists", I meant it's
a case the CL proposal handles implicitly.

bq. yeah, I feel it's inefficient to start a localizer process to only create symlinks..

No question. But if localizing a new resource takes a few seconds, for services that upgrade
over minutes/hours, then a few hundred milliseconds is not worth adding {{RUN_SCRIPT}} to
the CE.

> Support LinuxContainerExecutor to create symlinks for continuously localized resources
> --------------------------------------------------------------------------------------
>                 Key: YARN-5621
>                 URL: https://issues.apache.org/jira/browse/YARN-5621
>             Project: Hadoop YARN
>          Issue Type: Sub-task
>            Reporter: Jian He
>            Assignee: Jian He
>         Attachments: YARN-5621.1.patch, YARN-5621.2.patch, YARN-5621.3.patch, YARN-5621.4.patch,
> When new resources are localized, new symlink needs to be created for the localized resource.
This is the change for the LinuxContainerExecutor to create the symlinks.

This message was sent by Atlassian JIRA

To unsubscribe, e-mail: yarn-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: yarn-issues-help@hadoop.apache.org

View raw message