hadoop-mapreduce-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Sisu Xi <xis...@gmail.com>
Subject hadoop multinode, only master node doing the work
Date Sun, 13 Jul 2014 05:58:53 GMT
Hi, all:

I am new to hadoop. I followed the tutorial on
http://www.michael-noll.com/tutorials/running-hadoop-on-ubuntu-linux-multi-node-cluster/

and installed hadoop 2.2.0 on two 4-core Ubuntu 12.04 machines.

I can start the pi program, however, only the master node is doing the work
(I checked the top on each machine).
Seems the two nodes are configured correctly, because I can start  the
program in the slave node as well, and still only the master node is doing
the actual work.
I have tried different number of mappers for the pi program, and the
results is the same.

Is there anything else I can check?

In the end is my configure file on each host.

Thanks very much!

Sisu

---------yarn-site.xml-------

<property>
  <name>yarn.nodemanager.aux-services</name>
  <value>mapreduce_shuffle</value>
</property>


<property>
  <name>yarn.nodemanager.aux-services.mapreduce.shuffle.class</name>
  <value>org.apache.hadoop.mapred.ShuffleHandler</value>
</property>


<property>
  <name>yarn.resourcemanager.address</name>
  <value>master:8032</value>
</property>

---------------hdfs-site.xml--------------------

<property>
  <name>dfs.replication</name>
  <value>2</value>
</property>


<property>
  <name>dfs.namenode.name.dir</name>
  <value>file:/home/xisisu/mydata/hdfs/namenode</value>
</property>


<property>
  <name>dfs.datanode.data.dir</name>
  <value>file:/home/xisisu/mydata/hdfs/datanode</value>
</property>


-------------core-site.xml-------------

<property>
<name>fs.default.name</name>
<value>hdfs://master:9000</value>
</property>

------------------mapred-site.xml-----------------

<property>
  <name>mapreduce.framework.name</name>
  <value>yarn</value>
</property>

<property>
  <name>mapred.job.tracker</name>
  <value>master:54311</value>
  <description>The host and port that the MapReduce job tracker runs
  at.  If "local", then jobs are run in-process as a single map
  and reduce task.
  </description>
</property>




-- 


*Sisu Xi, PhD Candidate*

http://www.cse.wustl.edu/~xis/
Department of Computer Science and Engineering
Campus Box 1045
Washington University in St. Louis
One Brookings Drive
St. Louis, MO 63130

Mime
View raw message