From yarn-issues-return-125091-apmail-hadoop-yarn-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Sep 22 19:16:02 2017 Return-Path: X-Original-To: apmail-hadoop-yarn-issues-archive@minotaur.apache.org Delivered-To: apmail-hadoop-yarn-issues-archive@minotaur.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id D458810497 for ; Fri, 22 Sep 2017 19:16:02 +0000 (UTC) Received: (qmail 43295 invoked by uid 500); 22 Sep 2017 19:16:02 -0000 Delivered-To: apmail-hadoop-yarn-issues-archive@hadoop.apache.org Received: (qmail 43239 invoked by uid 500); 22 Sep 2017 19:16:02 -0000 Mailing-List: contact yarn-issues-help@hadoop.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Delivered-To: mailing list yarn-issues@hadoop.apache.org Received: (qmail 43228 invoked by uid 99); 22 Sep 2017 19:16:02 -0000 Received: from pnap-us-west-generic-nat.apache.org (HELO spamd1-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 22 Sep 2017 19:16:02 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd1-us-west.apache.org (ASF Mail Server at spamd1-us-west.apache.org) with ESMTP id E3BA1C29B4 for ; Fri, 22 Sep 2017 19:16:01 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd1-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: -99.202 X-Spam-Level: X-Spam-Status: No, score=-99.202 tagged_above=-999 required=6.31 tests=[KAM_ASCII_DIVIDERS=0.8, RP_MATCHES_RCVD=-0.001, SPF_PASS=-0.001, USER_IN_WHITELIST=-100] autolearn=disabled Received: from mx1-lw-us.apache.org ([10.40.0.8]) by localhost (spamd1-us-west.apache.org [10.40.0.7]) (amavisd-new, port 10024) with ESMTP id rqH0UXS8-iZK for ; Fri, 22 Sep 2017 19:16:01 +0000 (UTC) Received: from mailrelay1-us-west.apache.org (mailrelay1-us-west.apache.org [209.188.14.139]) by mx1-lw-us.apache.org (ASF Mail Server at mx1-lw-us.apache.org) with ESMTP id 3584A5F24F for ; Fri, 22 Sep 2017 19:16:01 +0000 (UTC) Received: from jira-lw-us.apache.org (unknown [207.244.88.139]) by mailrelay1-us-west.apache.org (ASF Mail Server at mailrelay1-us-west.apache.org) with ESMTP id B71ECE0D6A for ; Fri, 22 Sep 2017 19:16:00 +0000 (UTC) Received: from jira-lw-us.apache.org (localhost [127.0.0.1]) by jira-lw-us.apache.org (ASF Mail Server at jira-lw-us.apache.org) with ESMTP id 67E0B24211 for ; Fri, 22 Sep 2017 19:16:00 +0000 (UTC) Date: Fri, 22 Sep 2017 19:16:00 +0000 (UTC) From: "Eric Badger (JIRA)" To: yarn-issues@hadoop.apache.org Message-ID: In-Reply-To: References: Subject: [jira] [Commented] (YARN-7226) Whitelisted variables do not support delayed variable expansion MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394 [ https://issues.apache.org/jira/browse/YARN-7226?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16176961#comment-16176961 ] Eric Badger commented on YARN-7226: ----------------------------------- I can see the need for Docker containers to supply their own env vars in the image, but I agree that it should not change previous behavior of the whitelist. The client gets to decide what image to use, so they should know the layout of the image anyway. And if they don't, then they can fall back to the environment of the docker image. If we want for there to be a separate list of env vars that cannot be changed, then we can do that, but I think it should be empty and/or disabled by default to preserve current behavior. Otherwise, as [~jlowe] said, users won't be able to supply their own values for the env vars in the whitelist. So my vote would be for putting this patch in and then filing a followup jira to add a NM configurable list of variables that cannot be changed by the user (empty/disabled by default) > Whitelisted variables do not support delayed variable expansion > --------------------------------------------------------------- > > Key: YARN-7226 > URL: https://issues.apache.org/jira/browse/YARN-7226 > Project: Hadoop YARN > Issue Type: Bug > Components: nodemanager > Affects Versions: 2.9.0, 2.8.1, 3.0.0-alpha4 > Reporter: Jason Lowe > Assignee: Jason Lowe > Attachments: YARN-7226.001.patch, YARN-7226.002.patch > > > The nodemanager supports a configurable list of environment variables, via yarn.nodemanager.env-whitelist, that will be propagated to the container's environment unless those variables were specified in the container launch context. Unfortunately the handling of these whitelisted variables prevents using delayed variable expansion. For example, if a user shipped their own version of hadoop with their job via the distributed cache and specified: > {noformat} > HADOOP_COMMON_HOME={{PWD}}/my-private-hadoop/ > {noformat} > as part of their job, the variable will be set as the *literal* string: > {noformat} > $PWD/my-private-hadoop/ > {noformat} > rather than having $PWD expand to the container's current directory as it does for any other, non-whitelisted variable being set to the same value. -- This message was sent by Atlassian JIRA (v6.4.14#64029) --------------------------------------------------------------------- To unsubscribe, e-mail: yarn-issues-unsubscribe@hadoop.apache.org For additional commands, e-mail: yarn-issues-help@hadoop.apache.org