Mailing-List: contact user-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@hadoop.apache.org
Received-SPF: pass (athena.apache.org: domain of xisisu@gmail.com designates
 209.85.213.172 as permitted sender)
MIME-Version: 1.0
In-Reply-To: 
 <CAAs+fC4qBOzgAhjEKwe5GWMdJH0wqarShofXL7j=h=Bw6mf+xw@mail.gmail.com>
References: 
 <CAPqOm-r8Cavm=e4XOrGwgam8wmCR9MimJKDXO2itVVFxQBEiwg@mail.gmail.com>
 <CFE8262F.C104%Sambaiah_Kilaru@intuit.com>
 <CAPqOm-r_W26nwLdAoMk1usYk0HyU1r=WPBz8ssnLBireVrMMdw@mail.gmail.com>
 <CAPqOm-qW0g1RhH1+O9WM55rO8dJD7qOtK_X5h3K44VxMTcwbTw@mail.gmail.com>
 <CAAs+fC4qBOzgAhjEKwe5GWMdJH0wqarShofXL7j=h=Bw6mf+xw@mail.gmail.com>
From: Sisu Xi <xisisu@gmail.com>
Date: Sun, 13 Jul 2014 14:15:45 -0500
Message-ID: 
 <CAPqOm-p70sE3_nPP-PxOCp0TJ0s4FKqVQNjPds5gFggw6t52fQ@mail.gmail.com>
Subject: Re: hadoop multinode, only master node doing the work
To: user@hadoop.apache.org
Content-Type: multipart/alternative; boundary=089e0158a9aab8131104fe180264

--089e0158a9aab8131104fe180264
Content-Type: text/plain; charset=UTF-8

Hi, Ravi:

Yes! you are right! I did not realize there is a data dir on the slave node
as well. Thanks very much!
After deleting it and restart, now I can see the slave node.

However, when I ran the pi program (even from the slave node), still only
the master node is doing the work. The command I am using is:
xisisu@slave-01:/usr/local/hadoop$ hadoop jar
share/hadoop/mapreduce/hadoop-mapreduce-examples-2.2.0.jar pi 100 10000000

Attached is the output for hadoop dfsadmin -report

xisisu@slave-01:/usr/local/hadoop$ hadoop dfsadmin -report
DEPRECATED: Use of this script to execute hdfs command is deprecated.
Instead use the hdfs command for it.

OpenJDK 64-Bit Server VM warning: You have loaded library
/usr/local/hadoop/lib/native/libhadoop.so.1.0.0 which might have disabled
stack guard. The VM will try to fix the stack guard now.
It's highly recommended that you fix the library with 'execstack -c
<libfile>', or link it with '-z noexecstack'.
14/07/13 14:10:34 WARN util.NativeCodeLoader: Unable to load native-hadoop
library for your platform... using builtin-java classes where applicable
Configured Capacity: 28137644032 (26.21 GB)
Present Capacity: 16888445354 (15.73 GB)
DFS Remaining: 16887668736 (15.73 GB)
DFS Used: 776618 (758.42 KB)
DFS Used%: 0.00%
Under replicated blocks: 0
Blocks with corrupt replicas: 0
Missing blocks: 0

-------------------------------------------------
Datanodes available: 2 (2 total, 0 dead)

Live datanodes:
Name: 172.16.20.72:50010 (slave-01)
Hostname: slave-01
Decommission Status : Normal
Configured Capacity: 14068822016 (13.10 GB)
DFS Used: 388309 (379.21 KB)
Non DFS Used: 5623800619 (5.24 GB)
DFS Remaining: 8444633088 (7.86 GB)
DFS Used%: 0.00%
DFS Remaining%: 60.02%
Last contact: Sun Jul 13 14:10:34 CDT 2014


Name: 172.16.20.170:50010 (master)
Hostname: master
Decommission Status : Normal
Configured Capacity: 14068822016 (13.10 GB)
DFS Used: 388309 (379.21 KB)
Non DFS Used: 5625398059 (5.24 GB)
DFS Remaining: 8443035648 (7.86 GB)
DFS Used%: 0.00%
DFS Remaining%: 60.01%
Last contact: Sun Jul 13 14:10:34 CDT 2014


Sisu


On Sun, Jul 13, 2014 at 2:03 PM, Ravi Mutyala <ravi@hortonworks.com> wrote:

> Try cleaning up /home/xisisu/mydata/hdfs/datanode on the slave nodes
> andvstart  'datanode' on that again. That should work. If it doesn't, send
> the log message.
>
> Also see if nodemanagers are running on slaves nodes. If everything is
> running on master, then its not just HDFS issue, its also yarn nodemanagers
> not running.
>
> CONFIDENTIALITY NOTICE
> NOTICE: This message is intended for the use of the individual or entity
> to which it is addressed and may contain information that is confidential,
> privileged and exempt from disclosure under applicable law. If the reader
> of this message is not the intended recipient, you are hereby notified that
> any printing, copying, dissemination, distribution, disclosure or
> forwarding of this communication is strictly prohibited. If you have
> received this communication in error, please contact the sender immediately
> and delete it from your system. Thank You.


-- 


*Sisu Xi, PhD Candidate*

http://www.cse.wustl.edu/~xis/
Department of Computer Science and Engineering
Campus Box 1045
Washington University in St. Louis
One Brookings Drive
St. Louis, MO 63130

--089e0158a9aab8131104fe180264
Content-Type: text/html; charset=UTF-8
Content-Transfer-Encoding: quoted-printable

<div dir=3D"ltr">Hi, Ravi:<div><br></div><div>Yes! you are right! I did not=
 realize there is a data dir on the slave node as well. Thanks very much!</=
div><div>After deleting it and restart, now I can see the slave node.</div>

<div><br></div><div>However, when I ran the pi program (even from the slave=
 node), still only the master node is doing the work. The command I am usin=
g is:</div><div><div>xisisu@slave-01:/usr/local/hadoop$ hadoop jar share/ha=
doop/mapreduce/hadoop-mapreduce-examples-2.2.0.jar pi 100 10000000</div>

</div><div><br></div><div>Attached is the output for hadoop dfsadmin -repor=
t</div><div><br></div><div><div>xisisu@slave-01:/usr/local/hadoop$ hadoop d=
fsadmin -report</div><div>DEPRECATED: Use of this script to execute hdfs co=
mmand is deprecated.</div>

<div>Instead use the hdfs command for it.</div><div><br></div><div>OpenJDK =
64-Bit Server VM warning: You have loaded library /usr/local/hadoop/lib/nat=
ive/libhadoop.so.1.0.0 which might have disabled stack guard. The VM will t=
ry to fix the stack guard now.</div>

<div>It&#39;s highly recommended that you fix the library with &#39;execsta=
ck -c &lt;libfile&gt;&#39;, or link it with &#39;-z noexecstack&#39;.</div>=
<div>14/07/13 14:10:34 WARN util.NativeCodeLoader: Unable to load native-ha=
doop library for your platform... using builtin-java classes where applicab=
le</div>

<div>Configured Capacity: 28137644032 (26.21 GB)</div><div>Present Capacity=
: 16888445354 (15.73 GB)</div><div>DFS Remaining: 16887668736 (15.73 GB)</d=
iv><div>DFS Used: 776618 (758.42 KB)</div><div>DFS Used%: 0.00%</div><div>

Under replicated blocks: 0</div><div>Blocks with corrupt replicas: 0</div><=
div>Missing blocks: 0</div><div><br></div><div>----------------------------=
---------------------</div><div>Datanodes available: 2 (2 total, 0 dead)</d=
iv>

<div><br></div><div>Live datanodes:</div><div>Name: <a href=3D"http://172.1=
6.20.72:50010">172.16.20.72:50010</a> (slave-01)</div><div>Hostname: slave-=
01</div><div>Decommission Status : Normal</div><div>Configured Capacity: 14=
068822016 (13.10 GB)</div>

<div>DFS Used: 388309 (379.21 KB)</div><div>Non DFS Used: 5623800619 (5.24 =
GB)</div><div>DFS Remaining: 8444633088 (7.86 GB)</div><div>DFS Used%: 0.00=
%</div><div>DFS Remaining%: 60.02%</div><div>Last contact: Sun Jul 13 14:10=
:34 CDT 2014</div>

<div><br></div><div><br></div><div>Name: <a href=3D"http://172.16.20.170:50=
010">172.16.20.170:50010</a> (master)</div><div>Hostname: master</div><div>=
Decommission Status : Normal</div><div>Configured Capacity: 14068822016 (13=
.10 GB)</div>

<div>DFS Used: 388309 (379.21 KB)</div><div>Non DFS Used: 5625398059 (5.24 =
GB)</div><div>DFS Remaining: 8443035648 (7.86 GB)</div><div>DFS Used%: 0.00=
%</div><div>DFS Remaining%: 60.01%</div><div>Last contact: Sun Jul 13 14:10=
:34 CDT 2014</div>

</div><div><br></div><div><br></div><div><br></div><div>Sisu</div><div><br>=
</div><div><br></div></div><div class=3D"gmail_extra"><br><br><div class=3D=
"gmail_quote">On Sun, Jul 13, 2014 at 2:03 PM, Ravi Mutyala <span dir=3D"lt=
r">&lt;<a href=3D"mailto:ravi@hortonworks.com" target=3D"_blank">ravi@horto=
nworks.com</a>&gt;</span> wrote:<br>

<blockquote class=3D"gmail_quote" style=3D"margin:0 0 0 .8ex;border-left:1p=
x #ccc solid;padding-left:1ex"><div dir=3D"ltr">Try cleaning up /home/xisis=
u/mydata/hdfs/datanode on the slave nodes andvstart =C2=A0&#39;datanode&#39=
; on that again. That should work. If it doesn&#39;t, send the log message.=
<br>

<div>
<br>
Also see if nodemanagers are running on slaves nodes. If everything is runn=
ing on master, then its not just HDFS issue, its also yarn nodemanagers not=
 running.</div></div>

<br>
<span style=3D"color:rgb(128,128,128);font-family:Arial,sans-serif;font-siz=
e:10px">CONFIDENTIALITY NOTICE</span><br style=3D"color:rgb(128,128,128);fo=
nt-family:Arial,sans-serif;font-size:10px"><span style=3D"color:rgb(128,128=
,128);font-family:Arial,sans-serif;font-size:10px">NOTICE: This message is =
intended for the use of the individual or entity to which it is addressed a=
nd may contain information that is confidential, privileged and exempt from=
 disclosure under applicable law. If the reader of this message is not the =
intended recipient, you are hereby notified that any printing, copying, dis=
semination, distribution, disclosure or forwarding of this communication is=
 strictly prohibited. If you have received this communication in error, ple=
ase contact the sender immediately and delete it from your system. Thank Yo=
u.</span></blockquote>

</div><br><br clear=3D"all"><div><br></div>-- <br><div dir=3D"ltr"><div><im=
g src=3D"https://dl.dropboxusercontent.com/u/268069/Sisu_Contact.jpeg" widt=
h=3D"96" height=3D"96"><br></div><div><br></div><div><b>Sisu Xi, PhD Candid=
ate</b><br>

</div><br><a href=3D"http://www.cse.wustl.edu/~xis/" target=3D"_blank">http=
://www.cse.wustl.edu/~xis/</a><br>Department of Computer Science and Engine=
ering<br>Campus Box 1045<br>Washington University in St. Louis<br>One Brook=
ings Drive<br>

St. Louis, MO 63130</div>
</div>

--089e0158a9aab8131104fe180264--