hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Shane Kumpf (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (YARN-5736) YARN container executor config does not handle white space
Date Tue, 15 Nov 2016 17:44:59 GMT

    [ https://issues.apache.org/jira/browse/YARN-5736?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15667770#comment-15667770
] 

Shane Kumpf commented on YARN-5736:
-----------------------------------

Parsing of container-executor.cfg appears to have been impacted by this patch. If container-executor.cfg
contains entries with empty values, parsing doesn't work properly and all applications fail.

Here is the c-e output with additional debugging.
{code}
Failing this attempt.Diagnostics: Application application_1479220852002_0001 initialization
failed (exitCode=255) with output: read_config :Conf file name is : /usr/local/src/hadoop_install/hadoop/etc/hadoop/container-executor.cfg
read_config : Adding conf key : yarn.nodemanager.linux-container-executor.group
read_config : Adding conf value : yarn
read_config : Adding conf key : banned.users
read_config : Adding conf key : min.user.id
read_config : Adding conf value : 50
read_config : Adding conf key : allowed.system.users
read_config : Adding conf key : docker.binary
read_config : Adding conf value : /usr/bin/docker
read_config : Adding conf key : feature.docker.enabled
read_config : Adding conf value : 1
read_config : Adding conf key : feature.tc.enabled
read_config : Adding conf value : 1
Supplied key: yarn.nodemanager.linux-container-executor.group
Compare key: yarn.nodemanager.linux-container-executor.group
main : command provided 0
main : run as user is nobody
main : requested yarn user is root
get_value Supplied key: min.user.id
get_value Compare key: yarn.nodemanager.linux-container-executor.group
get_value Compare key: min.user.ids
get_value Compare key: docker.binarym.users
get_value Compare key: feature.docker.enabled
get_value Compare key: feature.tc.enabled
Requested user nobody is not whitelisted and has id 99,which is below the minimum allowed
1000
{code}

What was observed is that the min.user.id value is not properly parsed, because the key comparison
in {{configuration.c#get_value}} will never succeed. This is because the keys have extra characters
appended (min.user.id vs min.user.ids and docker.binary vs docker.binarym.users in the output
above). Keys with an "empty" value (represented by # in container-executor.cfg) are being
incorrectly combined with the key that follows in the config.

The relevant min.user.id parts below: 
{code}
-snip-
get_value Supplied key: min.user.id
-snip-
get_value Compare key: min.user.ids
{code}

Note that banned.users and min.user.id are combined and docker.binary and allowed.system.users
are combined.

Here is an example container-executor.cfg that shows the issue:
{code}
yarn.nodemanager.linux-container-executor.group=yarn
banned.users=#
min.user.id=50
allowed.system.users=#
docker.binary=/usr/bin/docker
feature.docker.enabled=1
feature.tc.enabled=1
{code}

I haven't quite tracked down the root cause, but reverting this patch does appear to resolve
the issue. I'm going to reopen this issue for further investigation.

> YARN container executor config does not handle white space
> ----------------------------------------------------------
>
>                 Key: YARN-5736
>                 URL: https://issues.apache.org/jira/browse/YARN-5736
>             Project: Hadoop YARN
>          Issue Type: Improvement
>          Components: nodemanager
>            Reporter: Miklos Szegedi
>            Assignee: Miklos Szegedi
>            Priority: Trivial
>              Labels: oct16-medium
>             Fix For: 3.0.0-alpha2
>
>         Attachments: YARN-5736.001.patch, YARN-5736.002.patch, YARN_5736.000.patch
>
>
> The container executor configuration reader does not handle white spaces or malformed
key value pairs in the config file correctly or gracefully
> as an example the following key value line which is part of the configuration (note the
<< is used as a marker to show the extra trailing space):
> yarn.nodemanager.linux-container-executor.group=yarn <<
> is a valid line but when you run the check over the file:
> [root@test]#./container-executor --checksetup
> Can't get group information for yarn - Success.
> [root@test]#
> It fails to find the yarn group but it really tries to find the "yarn " group which fails.
There is no trimming anywhere while processing the lines. If a space would be added in before
or after the = sign a failure would also occur.
> Minor nit is the fact that a failure still is logged as a Success



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: yarn-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: yarn-issues-help@hadoop.apache.org


Mime
View raw message