hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Shane Kumpf (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (YARN-5736) YARN container executor config does not handle white space
Date Tue, 15 Nov 2016 17:44:59 GMT

    [ https://issues.apache.org/jira/browse/YARN-5736?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15667770#comment-15667770

Shane Kumpf commented on YARN-5736:

Parsing of container-executor.cfg appears to have been impacted by this patch. If container-executor.cfg
contains entries with empty values, parsing doesn't work properly and all applications fail.

Here is the c-e output with additional debugging.
Failing this attempt.Diagnostics: Application application_1479220852002_0001 initialization
failed (exitCode=255) with output: read_config :Conf file name is : /usr/local/src/hadoop_install/hadoop/etc/hadoop/container-executor.cfg
read_config : Adding conf key : yarn.nodemanager.linux-container-executor.group
read_config : Adding conf value : yarn
read_config : Adding conf key : banned.users
read_config : Adding conf key : min.user.id
read_config : Adding conf value : 50
read_config : Adding conf key : allowed.system.users
read_config : Adding conf key : docker.binary
read_config : Adding conf value : /usr/bin/docker
read_config : Adding conf key : feature.docker.enabled
read_config : Adding conf value : 1
read_config : Adding conf key : feature.tc.enabled
read_config : Adding conf value : 1
Supplied key: yarn.nodemanager.linux-container-executor.group
Compare key: yarn.nodemanager.linux-container-executor.group
main : command provided 0
main : run as user is nobody
main : requested yarn user is root
get_value Supplied key: min.user.id
get_value Compare key: yarn.nodemanager.linux-container-executor.group
get_value Compare key: min.user.ids
get_value Compare key: docker.binarym.users
get_value Compare key: feature.docker.enabled
get_value Compare key: feature.tc.enabled
Requested user nobody is not whitelisted and has id 99,which is below the minimum allowed

What was observed is that the min.user.id value is not properly parsed, because the key comparison
in {{configuration.c#get_value}} will never succeed. This is because the keys have extra characters
appended (min.user.id vs min.user.ids and docker.binary vs docker.binarym.users in the output
above). Keys with an "empty" value (represented by # in container-executor.cfg) are being
incorrectly combined with the key that follows in the config.

The relevant min.user.id parts below: 
get_value Supplied key: min.user.id
get_value Compare key: min.user.ids

Note that banned.users and min.user.id are combined and docker.binary and allowed.system.users
are combined.

Here is an example container-executor.cfg that shows the issue:

I haven't quite tracked down the root cause, but reverting this patch does appear to resolve
the issue. I'm going to reopen this issue for further investigation.

> YARN container executor config does not handle white space
> ----------------------------------------------------------
>                 Key: YARN-5736
>                 URL: https://issues.apache.org/jira/browse/YARN-5736
>             Project: Hadoop YARN
>          Issue Type: Improvement
>          Components: nodemanager
>            Reporter: Miklos Szegedi
>            Assignee: Miklos Szegedi
>            Priority: Trivial
>              Labels: oct16-medium
>             Fix For: 3.0.0-alpha2
>         Attachments: YARN-5736.001.patch, YARN-5736.002.patch, YARN_5736.000.patch
> The container executor configuration reader does not handle white spaces or malformed
key value pairs in the config file correctly or gracefully
> as an example the following key value line which is part of the configuration (note the
<< is used as a marker to show the extra trailing space):
> yarn.nodemanager.linux-container-executor.group=yarn <<
> is a valid line but when you run the check over the file:
> [root@test]#./container-executor --checksetup
> Can't get group information for yarn - Success.
> [root@test]#
> It fails to find the yarn group but it really tries to find the "yarn " group which fails.
There is no trimming anywhere while processing the lines. If a space would be added in before
or after the = sign a failure would also occur.
> Minor nit is the fact that a failure still is logged as a Success

This message was sent by Atlassian JIRA

To unsubscribe, e-mail: yarn-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: yarn-issues-help@hadoop.apache.org

View raw message