hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Eric Yang (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (YARN-7654) Support ENTRY_POINT for docker container
Date Thu, 29 Mar 2018 23:22:00 GMT

    [ https://issues.apache.org/jira/browse/YARN-7654?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16419920#comment-16419920

Eric Yang commented on YARN-7654:

[~jlowe] . Thank you for the feedback.  The cmd file section based approach is the best option
in my opinion to ensure we don't have to look through multiple files to puzzle together the
launch command.

Rather than always adding WORK_DIR and then conditionally removing it, would it be better
to simply move the setup of WORK_DIR to the sanitizeEnv method? Then we only add it when we

WORK_DIR environment variable exists when JVM starts and we skip sanitizeEnv method because
we want what user passed in with minimized set of environment variables without expansion.
 This is the reason that I wrote it outside.  I agree on pushing the logic into sanitizeEnv
to improve code readability.  I will add subroutines to sanitizeEnv alternate path for entry
point mode.

Nit: This code still does a double-lookup and instead should cache the get result in a local.

Sorry, lost focus, will rewrite to:
boolean dockerOverride = false;
String tmpBuffer = environment
if (tmpBuffer != null) {
  dockerOverride = Bolean.parseBoolean(tmpBuffer);

I'm confused why the '-e ' flag destined for the docker run command is being encoded in DockerLinuxContainerRuntime
as part of the value payload for environment variables. I would expect the '-e' flag would
only be appearing in container-executor code as it parses the docker command file. Per the
--env-file suggestion above, I think we can ditch that flag entirely.

In version 006 and prior, this was true, but version 007 of the patch, -e flag encoding happens
only in container-executor docker utility method set_env.

Why is the detach behavior different for entry point containers vs. command override containers?

This is a good question.  In entry-point mode, the stdout/stderr are streamed directly from
docker run command to disk because the fork execvp clone the stdout/stderr directly to disk.
 Output is identical to what user would get from a terminal that docker errors and program
output are flush to the same buffer.  Bash mode launches the docker run to background, and
stdout/stderr are setup inside the docker container to write to bind-mounted location.  Hence,
Bash mode depends on another set of prelaunch log files to capture pre-launch tasks to capture
docker errors.  If Bash mode is not pushed to background both preloaunch.out and stdout.txt
will have almost identical output.  I keep bash mode as closer to existing implementation
as possible to prevent breakage.  These are the reasons that there are difference using foreground
mode for entry point mode, and detach for bash mode.

I will improve on the code style issues for all problems that you caught.  Thank you again
for the review.

> Support ENTRY_POINT for docker container
> ----------------------------------------
>                 Key: YARN-7654
>                 URL: https://issues.apache.org/jira/browse/YARN-7654
>             Project: Hadoop YARN
>          Issue Type: Sub-task
>          Components: yarn
>    Affects Versions: 3.1.0
>            Reporter: Eric Yang
>            Assignee: Eric Yang
>            Priority: Blocker
>         Attachments: YARN-7654.001.patch, YARN-7654.002.patch, YARN-7654.003.patch, YARN-7654.004.patch,
YARN-7654.005.patch, YARN-7654.006.patch, YARN-7654.007.patch
> Docker image may have ENTRY_POINT predefined, but this is not supported in the current
implementation.  It would be nice if we can detect existence of {{launch_command}} and base
on this variable launch docker container in different ways:
> h3. Launch command exists
> {code}
> docker run [image]:[version]
> docker exec [container_id] [launch_command]
> {code}
> {code}
> docker run [image]:[version]
> {code}

This message was sent by Atlassian JIRA

To unsubscribe, e-mail: yarn-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: yarn-issues-help@hadoop.apache.org

View raw message