hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Jason Lowe (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (YARN-6830) Support quoted strings for environment variables
Date Thu, 22 Mar 2018 15:57:00 GMT

    [ https://issues.apache.org/jira/browse/YARN-6830?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16409762#comment-16409762

Jason Lowe commented on YARN-6830:

Sorry to show up late on this.  I'm leaning towards [~aw]'s proposal to support separate properties
for environment variables in MapReduce.  To me it has the clearest path forward without the
possibility of breaking backwards compatibility.  We don't touch Configuration methods and
risk breaking any use cases that rely on the existing semantics, yet we can still support
environment variables with commas, quotes, or any other weird characters in their value. 
With YARN-5714 ordering the environment variables based on their inter-references, we shouldn't
need to rely on any magical ordering within a single property.  All that would remain is defining
the semantics of what happens when _both_ mapreduce.map.env and mapreduce.map.env.SOMEVAR
are defined and they conflict.  I lean towards the latter overriding any conflicting value
in the former.

Thoughts?  Objections?

> Support quoted strings for environment variables
> ------------------------------------------------
>                 Key: YARN-6830
>                 URL: https://issues.apache.org/jira/browse/YARN-6830
>             Project: Hadoop YARN
>          Issue Type: Bug
>            Reporter: Shane Kumpf
>            Assignee: Jim Brennan
>            Priority: Major
>         Attachments: YARN-6830.001.patch, YARN-6830.002.patch, YARN-6830.003.patch, YARN-6830.004.patch
> There are cases where it is necessary to allow for quoted string literals within environment
variables values when passed via the yarn command line interface.
> For example, consider the follow environment variables for a MR map task.
> {{MODE=bar}}
> {{IMAGE_NAME=foo}}
> {{MOUNTS=/tmp/foo,/tmp/bar}}
> When running the MR job, these environment variables are supplied as a comma delimited
> {{-Dmapreduce.map.env="MODE=bar,IMAGE_NAME=foo,MOUNTS=/tmp/foo,/tmp/bar"}}
> In this case, {{MOUNTS}} will be parsed and added to the task environment as {{MOUNTS=/tmp/foo}}.
Any attempts to quote the embedded comma separated value results in quote characters becoming
part of the value, and parsing still breaks down at the comma.
> This issue is to allow for quoting the comma separated value (escaped double or single
quote). This was mentioned on YARN-4595 and will impact YARN-5534 as well.

This message was sent by Atlassian JIRA

To unsubscribe, e-mail: yarn-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: yarn-issues-help@hadoop.apache.org

View raw message