hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Eric Yang (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (YARN-8384) stdout, stderr logs of a Native Service container is coming with group as nobody
Date Fri, 01 Jun 2018 00:23:00 GMT

    [ https://issues.apache.org/jira/browse/YARN-8384?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16497377#comment-16497377

Eric Yang commented on YARN-8384:

[~vinodkv], there are 3 different paths to start docker container:

1.  Someone who runs distributed shell that append:
 1> <LOG_DIR>/stdout 2> LOG_DIR>/stderr
Filename is user defined, but the file permission on the file depending on umask of the docker
image.  By default umask is 022, and anyone can read the file using other bit.  The file is
owned by uid:gid of the submission user in secure mode, or  nobody:nobody in insecure mode.
 This is a bit leaky by security standard.  Hadoop 3.1 implementation does not break the backward
compatibility for this mode.
2. Yarn Native Service yarn mode
This mode initializes stdout.txt and stderr.txt to uid of submission user, and gid of node
manager.  End user or viewing log file via node manager web application is the only two possible
users to look at the log.  If end user tries to add redirection of logs to other filename,
the generated file permission will end up as the docker container uid:gid and umask of the
docker container.  However, the output will end up in stdout.txt and stderr.txt because those
redirection are appended last in the launch command.

3. Yarn Service docker mode (ENTRY_POINT)
When using ENTRY_POINT, the stdout and stderr are written to stdout.txt and stderr.txt through
dup2 redirection.  It is not possible to use shell command redirection because the execution
is via execvp without shell expansion.  User can choose to write log to additional mount directories,
but the custom logs will not be aggregated by YARN framework.

Option 1 and 2 are kept around for backward compatibility reasons, but it is possible for
container to write file with permission that node manager can not process.  The setup of stdout.txt
and stderr.txt to owned by launching user, and readable by node manager in option 3 is safest
and recommended for future development.

> stdout, stderr logs of a Native Service container is coming with group as nobody
> --------------------------------------------------------------------------------
>                 Key: YARN-8384
>                 URL: https://issues.apache.org/jira/browse/YARN-8384
>             Project: Hadoop YARN
>          Issue Type: Sub-task
>          Components: yarn-native-services
>            Reporter: Sunil Govindan
>            Assignee: Eric Yang
>            Priority: Critical
>              Labels: docker
>         Attachments: YARN-8384.001.patch
> When {{yarn.nodemanager.linux-container-executor.nonsecure-mode.limit-users}} is set
to true, and {{yarn.nodemanager.linux-container-executor.nonsecure-mode.local-user}} is set
to nobody.
> This will cause the docker to run as nobody:nobody in yarn mode.
> The log files will be initialized as nobody:nobody:
> {noformat}
> rw-rr- 1 nobody hadoop 354 May 31 17:33 container-localizer-syslog
> rw-rr- 1 nobody hadoop 1042 May 31 17:35 directory.info
> rw-r---- 1 nobody hadoop 4944 May 31 17:35 launch_container.sh
> rw-rr- 1 nobody hadoop 440 May 31 17:35 prelaunch.err
> rw-rr- 1 nobody hadoop 100 May 31 17:35 prelaunch.out
> rw-r---- 1 nobody nobody 18733 May 31 17:37 stderr.txt
> rw-r---- 1 nobody nobody 400 May 31 17:35 stdout.txt
> {noformat}

This message was sent by Atlassian JIRA

To unsubscribe, e-mail: yarn-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: yarn-issues-help@hadoop.apache.org

View raw message