hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Jim Brennan (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (YARN-6830) Support quoted strings for environment variables
Date Wed, 21 Mar 2018 22:06:01 GMT

    [ https://issues.apache.org/jira/browse/YARN-6830?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16408656#comment-16408656
] 

Jim Brennan commented on YARN-6830:
-----------------------------------

There was some discussion of this at the latest docker sync-up meeting.  There was a slight
preference in that meeting for this over [YARN-8029], or the solution that [~aw] suggested
earlier:

{quote}
Rather than fight with a regex why not redefine the API instead?

When running the MR job, these environment variables are supplied as a comma delimited string.

-Dmapreduce.map.env="MODE=bar,IMAGE_NAME=foo,MOUNTS=/tmp/foo,/tmp/bar"

-Dmapreduce.map.env.MODE=bar
-Dmapreduce.map.env.IMAGE_NAME=foo
-Dmapreduce.map.env.MOUNTS=/tmp/foo,/tmp/bar

...

e.g, mapreduce.map.env.[foo]=bar gets turned into foo=bar

This greatly simplifies the input validation needed and makes it clear what is actually being
defined.
{quote}

This would be a more significant change, but since it would be orthogonal to the existing
behavior, it might be less risky than changing the regex.

So there are now three options on the table:
1 - [YARN-8029] - just use a different delimiter for the docker mount vars
2 - [YARN-6830] - this one, which changes the regex to allow commas inside a quoted string
(note that docker code needs to strip the quotes)
3 - Change the API to allow specifying environment variables as separate command line options
([~aw]'s proposal)



> Support quoted strings for environment variables
> ------------------------------------------------
>
>                 Key: YARN-6830
>                 URL: https://issues.apache.org/jira/browse/YARN-6830
>             Project: Hadoop YARN
>          Issue Type: Bug
>            Reporter: Shane Kumpf
>            Assignee: Jim Brennan
>            Priority: Major
>         Attachments: YARN-6830.001.patch, YARN-6830.002.patch, YARN-6830.003.patch, YARN-6830.004.patch
>
>
> There are cases where it is necessary to allow for quoted string literals within environment
variables values when passed via the yarn command line interface.
> For example, consider the follow environment variables for a MR map task.
> {{MODE=bar}}
> {{IMAGE_NAME=foo}}
> {{MOUNTS=/tmp/foo,/tmp/bar}}
> When running the MR job, these environment variables are supplied as a comma delimited
string.
> {{-Dmapreduce.map.env="MODE=bar,IMAGE_NAME=foo,MOUNTS=/tmp/foo,/tmp/bar"}}
> In this case, {{MOUNTS}} will be parsed and added to the task environment as {{MOUNTS=/tmp/foo}}.
Any attempts to quote the embedded comma separated value results in quote characters becoming
part of the value, and parsing still breaks down at the comma.
> This issue is to allow for quoting the comma separated value (escaped double or single
quote). This was mentioned on YARN-4595 and will impact YARN-5534 as well.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

---------------------------------------------------------------------
To unsubscribe, e-mail: yarn-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: yarn-issues-help@hadoop.apache.org


Mime
View raw message