hadoop-hdfs-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Allen Wittenauer"...@apache.org>
Subject Re: datanode not respecting HADOOP_CONF_DIR on 3.0.0-alpha2(?)
Date Fri, 29 Jul 2016 18:11:05 GMT


On 2016-07-29 03:05 (-0700), Ewan Higgs <Ewan.Higgs@hgst.com> wrote: 
> If HADOOP_CLASSPATH=$(hadoop classpath):/path/to/hadoop/tools/jar/dir then hdfs gets
very confused and somehow decides it won’t respect the HADOOP_CONF_DIR.
> 
> Solution: only add /path/to/hadoop/tools/jar/dir to HADOOP_CLASSPATH.

Short:

HADOOP_CLASSPATH is supplemental/additional.  The content of 'hadoop classpath' is always
present.  See the UnixShellGuide and hadoop-env.sh for more information, especially if you
are upgrading or accustomed to previous releases.

Long:

HADOOP_CONF_DIR is generally pushed onto the actual classpath first unless other directives
have been given elsewhere.  Since you're using trunk, you've got some additional tools to
figure out what is going on:

$ HADOOP_CLASSPATH=$(hadoop classpath):/tmp hdfs --debug version

This will spill out a bunch of info.  Key messages will be the system trying to de-dupe the
classpath (likely unsuccessfully due to shell expansion).  But the one you care most about
will be near the end. You'll get a line that says:

DEBUG: Final CLASSPATH:

The key thing there is that, at least in my test with a fresh install with the above command
line, the very first entry is my HADOOP_CONF_DIR.  So I'm not sure why HADOOP_CONF_DIR is
getting ignored in your case.  There might be other directives configured somewhere else that
may be causing the system confusion or who knows what all since the jars themselves include
*-default.xml to acts as the defaults.

Additionally, since you mentioned /path/to/hadoop/tools/jar/dir, if this is the actual tools
that ship with Hadoop, be aware that there are now much easier ways to add bits. You just
need to edit the HADOOP_OPTIONAL_TOOLS line in hadoop-env.sh to include the functionality
you want.  No need to even modify HADOOP_CLASSPATH anymore to add, e.g., S3 or Azure.



---------------------------------------------------------------------
To unsubscribe, e-mail: hdfs-dev-unsubscribe@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-help@hadoop.apache.org


Mime
View raw message