Mailing-List: contact user-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@hadoop.apache.org
Received-SPF: pass (athena.apache.org: domain of bhooshan.mogal@gmail.com
 designates 209.85.223.171 as permitted sender)
MIME-Version: 1.0
In-Reply-To: 
 <CAOEq2C5+PTCGTodkqmAKgUqJVQMyn68a2THibCVB-SNyriDgZg@mail.gmail.com>
References: 
 <CAOEq2C5+PTCGTodkqmAKgUqJVQMyn68a2THibCVB-SNyriDgZg@mail.gmail.com>
Date: Mon, 8 Sep 2014 14:40:03 -0700
Message-ID: 
 <CA+Di8eBtcdCJ0k+mb5fCAv6HQhevLCpMEOkVMMeOjWJqrNdGkg@mail.gmail.com>
Subject: Re: conf.get("dfs.data.dir") return null when hdfs-site.xml doesn't
 set it explicitly
From: Bhooshan Mogal <bhooshan.mogal@gmail.com>
To: "user@hadoop.apache.org" <user@hadoop.apache.org>
Content-Type: multipart/alternative; boundary=047d7bd757d06166e4050294a979

--047d7bd757d06166e4050294a979
Content-Type: text/plain; charset=UTF-8

Hi Demai,

When you read a property from the conf object, it will only have a value if
the conf object contains that property.

In your case, you created the conf object as new Configuration() -- adds
core-default and core-site.xml.

Then you added site.xmls (hdfs-site.xml and core-site.xml) from specific
locations. If none of these files have defined dfs.data.dir, then you will
get NULL. This is expected behavior.

What do you mean by retrieving the info from a live cluster? Even for
processes like datanode, namenode etc, the source of truth for these
properties is hdfs-site.xml. It is loaded from a specific location when you
start these services.

Question: Where are you running the above code? Is it on a node which has
other hadoop daemons as well?

My guess is that the path you are referring to (/etc/hadoop/conf.
cloudera.hdfs/core-site.xml) is not the right path where these config
properties are defined. Since this is a CDH cluster, you would probably be
best served by asking on the CDH mailing list as to where the right path to
these files is.


HTH,
Bhooshan


On Mon, Sep 8, 2014 at 11:47 AM, Demai Ni <nidmgg@gmail.com> wrote:

> hi, experts,
>
> I am trying to get the local filesystem directory of data node. My cluster
> is using CDH5.x (hadoop 2.3) and the default configuration. So the datanode
> is under file:///dfs/dn. I didn't specify the value in hdfs-site.xml.
>
> My code is something like:
>
> conf = new Configuration()
>
> // test both with and without the following two lines
> conf.addResource (new
> Path("/etc/hadoop/conf.cloudera.hdfs/hdfs-site.xml"));
> conf.addResource (new
> Path("/etc/hadoop/conf.cloudera.hdfs/core-site.xml"));
>
> // I also tried get("dfs.datanode.data.dir"), which also return NULL
> String dnDir = conf.get("dfs.data.dir");  // return NULL
>
> It looks like the get only look at the configuration file instead of
> retrieving the info from the live cluster?
>
> Many thanks for your help in advance.
>
> Demai
>


-- 
Bhooshan

--047d7bd757d06166e4050294a979
Content-Type: text/html; charset=UTF-8
Content-Transfer-Encoding: quoted-printable

<div dir=3D"ltr">Hi Demai,<div><br></div><div>When you read a property from=
 the conf object, it will only have a value if the conf object contains tha=
t property.</div><div><br></div><div>In your case, you created the conf obj=
ect as new Configuration() -- adds core-default and core-site.xml.</div><di=
v><br></div><div>Then you added site.xmls (hdfs-site.xml and core-site.xml)=
 from specific locations. If none of these files have defined dfs.data.dir,=
 then you will get NULL. This is expected behavior.=C2=A0</div><div><br></d=
iv><div>What do you mean by retrieving the info from a live cluster? Even f=
or processes like datanode, namenode etc, the source of truth for these pro=
perties is hdfs-site.xml. It is loaded from a specific location when you st=
art these services.</div><div><br></div><div>Question: Where are you runnin=
g the above code? Is it on a node which has other hadoop daemons as well?</=
div><div><br></div><div>My guess is that the path you are referring to (<sp=
an style=3D"font-size:13px;font-family:&#39;courier new&#39;,monospace">/et=
c/hadoop/conf.</span><span style=3D"font-size:13px;font-family:&#39;courier=
 new&#39;,monospace">cloudera.hdfs/core-site.xml</span>) is not the right p=
ath where these config properties are defined. Since this is a CDH cluster,=
 you would probably be best served by asking on the CDH mailing list as to =
where the right path to these files is.</div><div><br></div><div><br></div>=
<div>HTH,</div><div>Bhooshan</div></div><div class=3D"gmail_extra"><br><br>=
<div class=3D"gmail_quote">On Mon, Sep 8, 2014 at 11:47 AM, Demai Ni <span =
dir=3D"ltr">&lt;<a href=3D"mailto:nidmgg@gmail.com" target=3D"_blank">nidmg=
g@gmail.com</a>&gt;</span> wrote:<br><blockquote class=3D"gmail_quote" styl=
e=3D"margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div di=
r=3D"ltr"><div><div><div><div><div><div><div>hi, experts,<br><br></div>I am=
 trying to get the local filesystem directory of data node. My cluster is u=
sing CDH5.x (hadoop 2.3) and the default configuration. So the datanode is =
under file:///dfs/dn. I didn&#39;t specify the value in hdfs-site.xml. <br>=
<br></div>My code is something like: <br><span style=3D"font-family:courier=
 new,monospace"><br></span></div><span style=3D"font-family:courier new,mon=
ospace">conf =3D new Configuration()<br><br></span></div><div><span style=
=3D"font-family:courier new,monospace">// test both with and without the fo=
llowing two lines<br></span></div><span style=3D"font-family:courier new,mo=
nospace">conf.addResource (new Path(&quot;/etc/hadoop/conf.cloudera.hdfs/hd=
fs-site.xml&quot;)); <br>conf.addResource (new Path(&quot;/etc/hadoop/conf.=
cloudera.hdfs/core-site.xml&quot;));<br><br></span></div><div><span style=
=3D"font-family:courier new,monospace">// I also tried get(&quot;dfs.datano=
de.data.dir&quot;), which also return NULL<br></span></div><span style=3D"f=
ont-family:courier new,monospace">String dnDir =3D conf.get(&quot;dfs.data.=
dir&quot;);=C2=A0 // return NULL</span><br><br></div>It looks like the get =
only look at the configuration file instead of retrieving the info from the=
 live cluster? <br><br></div>Many thanks for your help in advance.<span cla=
ss=3D"HOEnZb"><font color=3D"#888888"><br><br>Demai<br></font></span></div>
</blockquote></div><br><br clear=3D"all"><div><br></div>-- <br><div dir=3D"=
ltr">Bhooshan</div>
</div>

--047d7bd757d06166e4050294a979--