Mailing-List: contact common-dev-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
Reply-To: common-dev@hadoop.apache.org
Content-Type: text/plain; charset=windows-1252
Mime-Version: 1.0 (Mac OS X Mail 7.3 \(1878.6\))
Subject: Re: use of HADOOP_HOME
From: Allen Wittenauer <aw@altiscale.com>
In-Reply-To: 
 <CADbBEnuP4Nm0=pNEACYf2nVy=dnNUHU4NoC91U0yGxiwUxkb+g@mail.gmail.com>
Date: Thu, 28 May 2015 10:13:49 -0700
Content-Transfer-Encoding: quoted-printable
Message-Id: <39AA035D-F917-435A-AD0A-AAA4CAE8519F@altiscale.com>
References: 
 <CADbBEnuP4Nm0=pNEACYf2nVy=dnNUHU4NoC91U0yGxiwUxkb+g@mail.gmail.com>
To: common-dev@hadoop.apache.org


On May 28, 2015, at 9:36 AM, Sangjin Lee <sjlee@apache.org> wrote:

> Hi folks,
>=20
> I noticed this while setting up a cluster based on the current trunk. =
It
> appears that setting HADOOP_HOME is now done much later (in
> hadoop_finalize) than branch-2. Importantly this is set *after*
> hadoop-env.sh (or yarn-env.sh) is invoked.
>=20
> In our version of hadoop-env.sh, we have used $HADOOP_HOME to define =
some
> more variables, but it appears that we can no longer rely on the
> HADOOP_HOME value in our *-env.sh customization. Is this an intended =
change
> in the recent shell script refactoring? What is the right thing to use =
in
> hadoop-env.sh for the location of hadoop?

	a) HADOOP_HOME was deprecated on Unix systems as part of (IIRC) =
0.21.  HADOOP_PREFIX was its replacement.  (No, I never understood the =
reasoning for this either.)  Past 0.21, it was never safe to rely upon =
HADOOP_HOME in *-env.sh files unless it is set prior to running the =
shell commands.

	b) That said, functionality-wise, HADOP_HOME is being set in =
pretty much the same place in the code flow.  *-env.sh has already been =
processed in both branch-2 and trunk by the time HADOOP_HOME is =
configured.  trunk only configures HADOOP_HOME for backward =
compatibility.  The rest of the code uses HADOOP_PREFIX as expected and =
very very early on the lifecycle. =20

	What you are likely seeing is the result of a bug fix:  trunk =
doesn=92t reprocess *-env.sh files when using the shin commands whereas =
branch-2 does it several times over. (This is also one of the reasons =
why Java command line options are duplicated too.)  So it likely worked =
for you because of this broken behavior.

	In my mind, it is a better practice to configure =
HADOOP_HOME/HADOOP_PREFIX outside of the *-env.sh files (e.g., =
/etc/profile.d on Linux) so that one can use them for PATH, etc.  That =
should guarantee expected behavior.