hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Allen Wittenauer (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (YARN-7011) yarn-daemon.sh is not respecting --config option
Date Thu, 17 Aug 2017 16:10:00 GMT

    [ https://issues.apache.org/jira/browse/YARN-7011?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16130744#comment-16130744

Allen Wittenauer commented on YARN-7011:

bq. I am not sure if everyone would agree that it is by design at least I am not convinced

As the person who designed and wrote the code in trunk, yes, it is working as intended.  

The problem is one of consistency.  There are a ton of different ways that HADOOP_CONF_DIR
can get set:  external vs. internal, --config vs. no option on CLI, HADOOP_CONF_DIR manual
vs. auto-discovered (2-3 ways), present in hadoop-env.sh vs. not, present but correct vs.
present and incorrect, etc ...  branch-2's behavior when going through this is matrix is wildly
unpredictable.  (and let's ignore's YARN_CONF_DIR because it just makes things worse and gets
us too far into the weeds)

Here's a fun experiment to show my point.  Unpack a new 2.8.1 tar ball.  Set HADOOP_CONF_DIR
in hadoop-2.8.1/etc/hadoop/hadoop-env.sh to something incorrect like /bin.  Now run hadoop

You'll find it starts out as "/bin".

Is that an incorrect behavior? 

On the one hand: yes, HADOOP_CONF_DIR is pointed to a wrong location.  But on the other: 
no, the user explicitly told the system that it wanted /bin as the configuration directory.
 branch-2 decided that in this instance /bin was correct and went forward.

Is that example different than when --config is used to point to a misconfigured directory?
Yes. branch-2's code decided that it was going to ignore what was in the configuration directory
as told to us by the user.

Should that behavior be different?

My argument is no:  the system should not suddenly decide the user was incorrect in one instance
vs. another. It absolutely must act predictably.  The user is expressly passing us a configuration
directory that tells us our configuration is actually in a different directory--very much
a misconfiguration.  The system accepts that as truth and moves on.

It's also worthwhile pointing out that 3.x adds a powerful wrinkle here:  .hadoop-env allows
the user to specify a different HADOOP_CONF_DIR, overwriting the installed one.  Here, redirection
is key.

If you don't like the current behavior, there are only two ways out.  (Let's be clear: I'll
reject any patch that continues branch-2's inconsistent behavior.)

* Option 1: always ignore HADOOP_CONF_DIR when set in hadoop-env.sh

If the code is reading hadoop-env.sh, then HADOOP_CONF_DIR is superfluous. It's only used
at that point is to give a different set of directories to act as replacements for the files
present in this one.

* Option 2: throw a warning

After reading hadoop-env.sh, detect if the value of HADOOP_CONF_DIR changed (after calculating
the full path to deal with symbolic links, etc.).  If it did, throw a warning up and then
either accept the new value (current behavior) or ignore it (see option 1).

> yarn-daemon.sh is not respecting --config option
> ------------------------------------------------
>                 Key: YARN-7011
>                 URL: https://issues.apache.org/jira/browse/YARN-7011
>             Project: Hadoop YARN
>          Issue Type: Bug
>          Components: yarn
>            Reporter: Sumana Sathish
>            Priority: Blocker
> Steps to reproduce:
> 1. Copy the conf to a temporary location /tmp/Conf
> 2. Modify anything in yarn-site.xml under /tmp/Conf/. Ex: Give invalid RM address
> 3. Restart the resourcemanager using yarn-daemon.sh using --config /tmp/Conf
> 4. --config is not respected as the changes made in /tmp/Conf/yarn-site.xml is not taken
in while restarting RM

This message was sent by Atlassian JIRA

To unsubscribe, e-mail: yarn-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: yarn-issues-help@hadoop.apache.org

View raw message