From yarn-issues-return-146568-archive-asf-public=cust-asf.ponee.io@hadoop.apache.org Fri Jun 1 02:23:05 2018 Return-Path: X-Original-To: archive-asf-public@cust-asf.ponee.io Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by mx-eu-01.ponee.io (Postfix) with SMTP id A2044180632 for ; Fri, 1 Jun 2018 02:23:04 +0200 (CEST) Received: (qmail 48435 invoked by uid 500); 1 Jun 2018 00:23:03 -0000 Mailing-List: contact yarn-issues-help@hadoop.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Delivered-To: mailing list yarn-issues@hadoop.apache.org Received: (qmail 48423 invoked by uid 99); 1 Jun 2018 00:23:03 -0000 Received: from pnap-us-west-generic-nat.apache.org (HELO spamd2-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 01 Jun 2018 00:23:03 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd2-us-west.apache.org (ASF Mail Server at spamd2-us-west.apache.org) with ESMTP id 504651A2C42 for ; Fri, 1 Jun 2018 00:23:03 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd2-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: -109.501 X-Spam-Level: X-Spam-Status: No, score=-109.501 tagged_above=-999 required=6.31 tests=[ENV_AND_HDR_SPF_MATCH=-0.5, KAM_ASCII_DIVIDERS=0.8, RCVD_IN_DNSWL_MED=-2.3, SPF_PASS=-0.001, USER_IN_DEF_SPF_WL=-7.5, USER_IN_WHITELIST=-100] autolearn=disabled Received: from mx1-lw-eu.apache.org ([10.40.0.8]) by localhost (spamd2-us-west.apache.org [10.40.0.9]) (amavisd-new, port 10024) with ESMTP id 10gem5eE786p for ; Fri, 1 Jun 2018 00:23:01 +0000 (UTC) Received: from mailrelay1-us-west.apache.org (mailrelay1-us-west.apache.org [209.188.14.139]) by mx1-lw-eu.apache.org (ASF Mail Server at mx1-lw-eu.apache.org) with ESMTP id 253B45F470 for ; Fri, 1 Jun 2018 00:23:01 +0000 (UTC) Received: from jira-lw-us.apache.org (unknown [207.244.88.139]) by mailrelay1-us-west.apache.org (ASF Mail Server at mailrelay1-us-west.apache.org) with ESMTP id 59C2AE0016 for ; Fri, 1 Jun 2018 00:23:00 +0000 (UTC) Received: from jira-lw-us.apache.org (localhost [127.0.0.1]) by jira-lw-us.apache.org (ASF Mail Server at jira-lw-us.apache.org) with ESMTP id 1846221094 for ; Fri, 1 Jun 2018 00:23:00 +0000 (UTC) Date: Fri, 1 Jun 2018 00:23:00 +0000 (UTC) From: "Eric Yang (JIRA)" To: yarn-issues@hadoop.apache.org Message-ID: In-Reply-To: References: Subject: [jira] [Commented] (YARN-8384) stdout, stderr logs of a Native Service container is coming with group as nobody MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394 [ https://issues.apache.org/jira/browse/YARN-8384?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16497377#comment-16497377 ] Eric Yang commented on YARN-8384: --------------------------------- [~vinodkv], there are 3 different paths to start docker container: 1. Someone who runs distributed shell that append: {code} 1> /stdout 2> LOG_DIR>/stderr {code} Filename is user defined, but the file permission on the file depending on umask of the docker image. By default umask is 022, and anyone can read the file using other bit. The file is owned by uid:gid of the submission user in secure mode, or nobody:nobody in insecure mode. This is a bit leaky by security standard. Hadoop 3.1 implementation does not break the backward compatibility for this mode. 2. Yarn Native Service yarn mode This mode initializes stdout.txt and stderr.txt to uid of submission user, and gid of node manager. End user or viewing log file via node manager web application is the only two possible users to look at the log. If end user tries to add redirection of logs to other filename, the generated file permission will end up as the docker container uid:gid and umask of the docker container. However, the output will end up in stdout.txt and stderr.txt because those redirection are appended last in the launch command. 3. Yarn Service docker mode (ENTRY_POINT) When using ENTRY_POINT, the stdout and stderr are written to stdout.txt and stderr.txt through dup2 redirection. It is not possible to use shell command redirection because the execution is via execvp without shell expansion. User can choose to write log to additional mount directories, but the custom logs will not be aggregated by YARN framework. Option 1 and 2 are kept around for backward compatibility reasons, but it is possible for container to write file with permission that node manager can not process. The setup of stdout.txt and stderr.txt to owned by launching user, and readable by node manager in option 3 is safest and recommended for future development. > stdout, stderr logs of a Native Service container is coming with group as nobody > -------------------------------------------------------------------------------- > > Key: YARN-8384 > URL: https://issues.apache.org/jira/browse/YARN-8384 > Project: Hadoop YARN > Issue Type: Sub-task > Components: yarn-native-services > Reporter: Sunil Govindan > Assignee: Eric Yang > Priority: Critical > Labels: docker > Attachments: YARN-8384.001.patch > > > When {{yarn.nodemanager.linux-container-executor.nonsecure-mode.limit-users}} is set to true, and {{yarn.nodemanager.linux-container-executor.nonsecure-mode.local-user}} is set to nobody. > This will cause the docker to run as nobody:nobody in yarn mode. > The log files will be initialized as nobody:nobody: > {noformat} > rw-rr- 1 nobody hadoop 354 May 31 17:33 container-localizer-syslog > rw-rr- 1 nobody hadoop 1042 May 31 17:35 directory.info > rw-r---- 1 nobody hadoop 4944 May 31 17:35 launch_container.sh > rw-rr- 1 nobody hadoop 440 May 31 17:35 prelaunch.err > rw-rr- 1 nobody hadoop 100 May 31 17:35 prelaunch.out > rw-r---- 1 nobody nobody 18733 May 31 17:37 stderr.txt > rw-r---- 1 nobody nobody 400 May 31 17:35 stdout.txt > {noformat} -- This message was sent by Atlassian JIRA (v7.6.3#76005) --------------------------------------------------------------------- To unsubscribe, e-mail: yarn-issues-unsubscribe@hadoop.apache.org For additional commands, e-mail: yarn-issues-help@hadoop.apache.org