Mailing-List: contact yarn-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
Date: Thu, 17 Aug 2017 16:10:00 +0000 (UTC)
From: "Allen Wittenauer (JIRA)" <jira@apache.org>
To: yarn-issues@hadoop.apache.org
Message-ID: <JIRA.13094635.1502752815000.55515.1502986200211@Atlassian.JIRA>
In-Reply-To: <JIRA.13094635.1502752815000@Atlassian.JIRA>
References: <JIRA.13094635.1502752815000@Atlassian.JIRA> <JIRA.13094635.1502752815962@jira-lw-us.apache.org>
Subject: [jira] [Commented] (YARN-7011) yarn-daemon.sh is not respecting
 --config option
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
archived-at: Thu, 17 Aug 2017 16:10:06 -0000


    [ https://issues.apache.org/jira/browse/YARN-7011?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16130744#comment-16130744 ] 

Allen Wittenauer commented on YARN-7011:
----------------------------------------

bq. I am not sure if everyone would agree that it is by design at least I am not convinced yet.

As the person who designed and wrote the code in trunk, yes, it is working as intended.  

The problem is one of consistency.  There are a ton of different ways that HADOOP_CONF_DIR can get set:  external vs. internal, --config vs. no option on CLI, HADOOP_CONF_DIR manual vs. auto-discovered (2-3 ways), present in hadoop-env.sh vs. not, present but correct vs. present and incorrect, etc ...  branch-2's behavior when going through this is matrix is wildly unpredictable.  (and let's ignore's YARN_CONF_DIR because it just makes things worse and gets us too far into the weeds)

Here's a fun experiment to show my point.  Unpack a new 2.8.1 tar ball.  Set HADOOP_CONF_DIR in hadoop-2.8.1/etc/hadoop/hadoop-env.sh to something incorrect like /bin.  Now run hadoop classpath.

You'll find it starts out as "/bin".

Is that an incorrect behavior? 

On the one hand: yes, HADOOP_CONF_DIR is pointed to a wrong location.  But on the other:  no, the user explicitly told the system that it wanted /bin as the configuration directory.  branch-2 decided that in this instance /bin was correct and went forward.

Is that example different than when --config is used to point to a misconfigured directory? Yes. branch-2's code decided that it was going to ignore what was in the configuration directory as told to us by the user.

Should that behavior be different?

My argument is no:  the system should not suddenly decide the user was incorrect in one instance vs. another. It absolutely must act predictably.  The user is expressly passing us a configuration directory that tells us our configuration is actually in a different directory--very much a misconfiguration.  The system accepts that as truth and moves on.

It's also worthwhile pointing out that 3.x adds a powerful wrinkle here:  .hadoop-env allows the user to specify a different HADOOP_CONF_DIR, overwriting the installed one.  Here, redirection is key.

If you don't like the current behavior, there are only two ways out.  (Let's be clear: I'll reject any patch that continues branch-2's inconsistent behavior.)

* Option 1: always ignore HADOOP_CONF_DIR when set in hadoop-env.sh

If the code is reading hadoop-env.sh, then HADOOP_CONF_DIR is superfluous. It's only used at that point is to give a different set of directories to act as replacements for the files present in this one.

* Option 2: throw a warning

After reading hadoop-env.sh, detect if the value of HADOOP_CONF_DIR changed (after calculating the full path to deal with symbolic links, etc.).  If it did, throw a warning up and then either accept the new value (current behavior) or ignore it (see option 1).

> yarn-daemon.sh is not respecting --config option
> ------------------------------------------------
>
>                 Key: YARN-7011
>                 URL: https://issues.apache.org/jira/browse/YARN-7011
>             Project: Hadoop YARN
>          Issue Type: Bug
>          Components: yarn
>            Reporter: Sumana Sathish
>            Priority: Blocker
>
> Steps to reproduce:
> 1. Copy the conf to a temporary location /tmp/Conf
> 2. Modify anything in yarn-site.xml under /tmp/Conf/. Ex: Give invalid RM address
> 3. Restart the resourcemanager using yarn-daemon.sh using --config /tmp/Conf
> 4. --config is not respected as the changes made in /tmp/Conf/yarn-site.xml is not taken in while restarting RM


--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

---------------------------------------------------------------------
To unsubscribe, e-mail: yarn-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: yarn-issues-help@hadoop.apache.org