hadoop-hdfs-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Sisu Xi <xis...@gmail.com>
Subject Re: hadoop multinode, only master node doing the work
Date Sun, 13 Jul 2014 17:53:09 GMT
Hi, Sam:

Thanks for your help! You are right, seems only one node is running.
Here is the output:

xisisu@slave-01:/usr/local/hadoop$ hadoop dfsadmin -report
DEPRECATED: Use of this script to execute hdfs command is deprecated.
Instead use the hdfs command for it.

OpenJDK 64-Bit Server VM warning: You have loaded library
/usr/local/hadoop/lib/native/libhadoop.so.1.0.0 which might have disabled
stack guard. The VM will try to fix the stack guard now.
It's highly recommended that you fix the library with 'execstack -c
<libfile>', or link it with '-z noexecstack'.
14/07/13 12:50:14 WARN util.NativeCodeLoader: Unable to load native-hadoop
library for your platform... using builtin-java classes where applicable
Configured Capacity: 14068822016 (13.10 GB)
Present Capacity: 8425232425 (7.85 GB)
DFS Remaining: 8423534592 (7.85 GB)
DFS Used: 1697833 (1.62 MB)
DFS Used%: 0.02%
Under replicated blocks: 21
Blocks with corrupt replicas: 0
Missing blocks: 0

-------------------------------------------------
Datanodes available: 1 (1 total, 0 dead)

Live datanodes:
Name: 172.16.20.170:50010 (master)
Hostname: master
Decommission Status : Normal
Configured Capacity: 14068822016 (13.10 GB)
DFS Used: 1697833 (1.62 MB)
Non DFS Used: 5643589591 (5.26 GB)
DFS Remaining: 8423534592 (7.85 GB)
DFS Used%: 0.01%
DFS Remaining%: 59.87%
Last contact: Sun Jul 13 12:50:13 CDT 2014


I get the same output when I run the command on the slave node.


Is there anything I am missing in the config file?

Thanks very much!

Sisu



On Sun, Jul 13, 2014 at 1:36 AM, Kilaru, Sambaiah <
Sambaiah_Kilaru@intuit.com> wrote:

>  HI Sisu Xi,
>
>  On the master node can you check
> Hadoop dfsadmin –report
> And listing all the slave nodes or you can check master URL and it should
> all datanodes listed as slave nodes.
> Check for RM UI and slave nodes listed there also.
>
>  Thanks,
> Sam
>
>   From: Sisu Xi <xisisu@gmail.com>
> Reply-To: "user@hadoop.apache.org" <user@hadoop.apache.org>
> Date: Sunday, July 13, 2014 at 11:28 AM
> To: "user@hadoop.apache.org" <user@hadoop.apache.org>
> Subject: hadoop multinode, only master node doing the work
>
>   Hi, all:
>
>  I am new to hadoop. I followed the tutorial on
>
> http://www.michael-noll.com/tutorials/running-hadoop-on-ubuntu-linux-multi-node-cluster/
>
>  and installed hadoop 2.2.0 on two 4-core Ubuntu 12.04 machines.
>
>  I can start the pi program, however, only the master node is doing the
> work (I checked the top on each machine).
> Seems the two nodes are configured correctly, because I can start  the
> program in the slave node as well, and still only the master node is doing
> the actual work.
> I have tried different number of mappers for the pi program, and the
> results is the same.
>
>  Is there anything else I can check?
>
>  In the end is my configure file on each host.
>
>  Thanks very much!
>
>  Sisu
>
>  ---------yarn-site.xml-------
>
>  <property>
>   <name>yarn.nodemanager.aux-services</name>
>   <value>mapreduce_shuffle</value>
> </property>
>
>
>  <property>
>   <name>yarn.nodemanager.aux-services.mapreduce.shuffle.class</name>
>   <value>org.apache.hadoop.mapred.ShuffleHandler</value>
> </property>
>
>
>  <property>
>   <name>yarn.resourcemanager.address</name>
>   <value>master:8032</value>
> </property>
>
>  ---------------hdfs-site.xml--------------------
>
>  <property>
>   <name>dfs.replication</name>
>   <value>2</value>
> </property>
>
>
>  <property>
>   <name>dfs.namenode.name.dir</name>
>   <value>file:/home/xisisu/mydata/hdfs/namenode</value>
> </property>
>
>
>  <property>
>   <name>dfs.datanode.data.dir</name>
>   <value>file:/home/xisisu/mydata/hdfs/datanode</value>
> </property>
>
>
>  -------------core-site.xml-------------
>
>  <property>
>  <name>fs.default.name</name>
> <value>hdfs://master:9000</value>
> </property>
>
>  ------------------mapred-site.xml-----------------
>
>  <property>
>   <name>mapreduce.framework.name</name>
>   <value>yarn</value>
> </property>
>
>  <property>
>   <name>mapred.job.tracker</name>
>   <value>master:54311</value>
>   <description>The host and port that the MapReduce job tracker runs
>   at.  If "local", then jobs are run in-process as a single map
>   and reduce task.
>   </description>
> </property>
>
>
>
>
>  --
>
>
>  *Sisu Xi, PhD Candidate*
>
> http://www.cse.wustl.edu/~xis/
> Department of Computer Science and Engineering
> Campus Box 1045
> Washington University in St. Louis
> One Brookings Drive
> St. Louis, MO 63130
>



-- 


*Sisu Xi, PhD Candidate*

http://www.cse.wustl.edu/~xis/
Department of Computer Science and Engineering
Campus Box 1045
Washington University in St. Louis
One Brookings Drive
St. Louis, MO 63130

Mime
View raw message