hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Shane Kumpf (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (YARN-6830) Support quoted strings for environment variables
Date Thu, 22 Mar 2018 16:31:00 GMT

    [ https://issues.apache.org/jira/browse/YARN-6830?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16409822#comment-16409822

Shane Kumpf commented on YARN-6830:

Thanks for taking this over and the patch [~Jim_Brennan]!

{quote}I'm leaning towards Allen Wittenauer's proposal to support separate properties for
environment variables in MapReduce.{quote}
I am also in favor of this approach for the reasons [~jlowe] mentions and to avoid risky regex

 All that would remain is defining the semantics of what happens when both mapreduce.map.env
and mapreduce.map.env.SOMEVAR are defined and they conflict. I lean towards the latter overriding
any conflicting value in the former.
Initially I was thinking the opposite to maintain backwards compatibility, however, support
for mapreduce.map.env.SOMEVAR would be a new feature that a user would need to opt in to using.
If the user has gone out of their way to add this new configuration, it seems likely that
they want the new value to be used, so I tend to agree with the approach you outlined [~jlowe].

> Support quoted strings for environment variables
> ------------------------------------------------
>                 Key: YARN-6830
>                 URL: https://issues.apache.org/jira/browse/YARN-6830
>             Project: Hadoop YARN
>          Issue Type: Bug
>            Reporter: Shane Kumpf
>            Assignee: Jim Brennan
>            Priority: Major
>         Attachments: YARN-6830.001.patch, YARN-6830.002.patch, YARN-6830.003.patch, YARN-6830.004.patch
> There are cases where it is necessary to allow for quoted string literals within environment
variables values when passed via the yarn command line interface.
> For example, consider the follow environment variables for a MR map task.
> {{MODE=bar}}
> {{IMAGE_NAME=foo}}
> {{MOUNTS=/tmp/foo,/tmp/bar}}
> When running the MR job, these environment variables are supplied as a comma delimited
> {{-Dmapreduce.map.env="MODE=bar,IMAGE_NAME=foo,MOUNTS=/tmp/foo,/tmp/bar"}}
> In this case, {{MOUNTS}} will be parsed and added to the task environment as {{MOUNTS=/tmp/foo}}.
Any attempts to quote the embedded comma separated value results in quote characters becoming
part of the value, and parsing still breaks down at the comma.
> This issue is to allow for quoting the comma separated value (escaped double or single
quote). This was mentioned on YARN-4595 and will impact YARN-5534 as well.

This message was sent by Atlassian JIRA

To unsubscribe, e-mail: yarn-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: yarn-issues-help@hadoop.apache.org

View raw message