hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Chris Douglas (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (YARN-5621) Support LinuxContainerExecutor to create symlinks for continuously localized resources
Date Fri, 09 Sep 2016 23:07:20 GMT

    [ https://issues.apache.org/jira/browse/YARN-5621?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15478543#comment-15478543

Chris Douglas commented on YARN-5621:

bq. FWIW, I'd love to see us drop the container launch script. I haven't tried it, but I suspect
we can do lots of fun things with the env vars.

For all containers, we have (1) NM constants (2) some user args we verify (e.g., container
ID matches the token, is correctly formatted, etc.) used as args to the CE (which should validate
that each of these args conforms to a schema). These are the args used to build paths. All
other args (3) the user can control should be written to the container launch script, which
is executed with the same permissions the container would have. The intent was to have all
quoting games happen after we've switched to the user's context, and after we've discarded
the NM environment. The implementation may have gaps, but is there a problem with the concept?

This JIRA follows a similar pattern, but without validation of args in the CE. If it were
restricted s.t. the source had a fixed format in {{nmPrivate}} and the destination was derived
from a formatted {{ContainerID}}, it could have comparable guarantees as the container start.

Unless the resource is public, could this avoid modifying the CE by moving the symlink to
the {{ContainerLocalizer}}? It could receive a symlink command on a heartbeat, it's already
running as the user, it may already be running to download the resource...

> Support LinuxContainerExecutor to create symlinks for continuously localized resources
> --------------------------------------------------------------------------------------
>                 Key: YARN-5621
>                 URL: https://issues.apache.org/jira/browse/YARN-5621
>             Project: Hadoop YARN
>          Issue Type: Sub-task
>            Reporter: Jian He
>            Assignee: Jian He
>         Attachments: YARN-5621.1.patch, YARN-5621.2.patch, YARN-5621.3.patch
> When new resources are localized, new symlink needs to be created for the localized resource.
This is the change for the LinuxContainerExecutor to create the symlinks.

This message was sent by Atlassian JIRA

To unsubscribe, e-mail: yarn-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: yarn-issues-help@hadoop.apache.org

View raw message