Return-Path: 
 <user-return-16123-apmail-hadoop-mapreduce-user-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-mapreduce-user-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-mapreduce-user-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 021131179E
	for <apmail-hadoop-mapreduce-user-archive@minotaur.apache.org>;
 Sun, 13 Jul 2014 06:00:11 +0000 (UTC)
Received: (qmail 77191 invoked by uid 500); 13 Jul 2014 06:00:01 -0000
Delivered-To: apmail-hadoop-mapreduce-user-archive@hadoop.apache.org
Received: (qmail 77054 invoked by uid 500); 13 Jul 2014 06:00:01 -0000
Mailing-List: contact user-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:user-help@hadoop.apache.org>
List-Unsubscribe: <mailto:user-unsubscribe@hadoop.apache.org>
List-Post: <mailto:user@hadoop.apache.org>
List-Id: <user.hadoop.apache.org>
Reply-To: user@hadoop.apache.org
Delivered-To: mailing list user@hadoop.apache.org
Received: (qmail 77044 invoked by uid 99); 13 Jul 2014 06:00:00 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 13 Jul 2014 06:00:00 +0000
X-ASF-Spam-Status: No, hits=1.5 required=5.0
	tests=HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of xisisu@gmail.com designates
 209.85.213.173 as permitted sender)
Received: from [209.85.213.173] (HELO mail-ig0-f173.google.com)
 (209.85.213.173)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 13 Jul 2014 05:59:59 +0000
Received: by mail-ig0-f173.google.com with SMTP id h18so758579igc.12
        for <user@hadoop.apache.org>; Sat, 12 Jul 2014 22:59:34 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=20120113;
        h=mime-version:from:date:message-id:subject:to:content-type;
        bh=M2blX1A4r2O2kiCSB4f8XCc0nEL6WnYqCH3cdTT1yTk=;
        b=EeRGUQVm1c3zzY4LJ2h8Zw0XOm5s/r1KXS1uerISVijoCt+08bec3o//7K6IoPJhBe
         E+DZjdesI2G6fmC4x95U3cgP8+5nxFDl8YsyIBfwBUnJGm9Zhm1MmVlhJSrX/q5AX/5y
         mx1oA0DIgp7Sb/ixXiZ04JvRWC0QQEqWjChRp2RIHeYpmKSTS4fxvMWlWdaln+Kj47Mq
         yWCQuchoylDkGELZXye5jciFd+b9eILdCB6H0etnLzSGaE0z20E4oJjy7BJ7DtPZd0Jk
         hgjs/va3ZRxewqGtAtL2uliKJj8I+nBTmyh/ZCXwH5ML6gos9N957EFQQrYg2EJ5nc5L
         u3HQ==
X-Received: by 10.50.164.201 with SMTP id ys9mr16200815igb.40.1405231174070;
 Sat, 12 Jul 2014 22:59:34 -0700 (PDT)
MIME-Version: 1.0
Received: by 10.64.81.169 with HTTP; Sat, 12 Jul 2014 22:58:53 -0700 (PDT)
From: Sisu Xi <xisisu@gmail.com>
Date: Sun, 13 Jul 2014 00:58:53 -0500
Message-ID: 
 <CAPqOm-r8Cavm=e4XOrGwgam8wmCR9MimJKDXO2itVVFxQBEiwg@mail.gmail.com>
Subject: hadoop multinode, only master node doing the work
To: user@hadoop.apache.org
Content-Type: multipart/alternative; boundary=089e0149c1def5375404fe0ce05a
X-Virus-Checked: Checked by ClamAV on apache.org

--089e0149c1def5375404fe0ce05a
Content-Type: text/plain; charset=UTF-8

Hi, all:

I am new to hadoop. I followed the tutorial on
http://www.michael-noll.com/tutorials/running-hadoop-on-ubuntu-linux-multi-node-cluster/

and installed hadoop 2.2.0 on two 4-core Ubuntu 12.04 machines.

I can start the pi program, however, only the master node is doing the work
(I checked the top on each machine).
Seems the two nodes are configured correctly, because I can start  the
program in the slave node as well, and still only the master node is doing
the actual work.
I have tried different number of mappers for the pi program, and the
results is the same.

Is there anything else I can check?

In the end is my configure file on each host.

Thanks very much!

Sisu

---------yarn-site.xml-------

<property>
  <name>yarn.nodemanager.aux-services</name>
  <value>mapreduce_shuffle</value>
</property>


<property>
  <name>yarn.nodemanager.aux-services.mapreduce.shuffle.class</name>
  <value>org.apache.hadoop.mapred.ShuffleHandler</value>
</property>


<property>
  <name>yarn.resourcemanager.address</name>
  <value>master:8032</value>
</property>

---------------hdfs-site.xml--------------------

<property>
  <name>dfs.replication</name>
  <value>2</value>
</property>


<property>
  <name>dfs.namenode.name.dir</name>
  <value>file:/home/xisisu/mydata/hdfs/namenode</value>
</property>


<property>
  <name>dfs.datanode.data.dir</name>
  <value>file:/home/xisisu/mydata/hdfs/datanode</value>
</property>


-------------core-site.xml-------------

<property>
<name>fs.default.name</name>
<value>hdfs://master:9000</value>
</property>

------------------mapred-site.xml-----------------

<property>
  <name>mapreduce.framework.name</name>
  <value>yarn</value>
</property>

<property>
  <name>mapred.job.tracker</name>
  <value>master:54311</value>
  <description>The host and port that the MapReduce job tracker runs
  at.  If "local", then jobs are run in-process as a single map
  and reduce task.
  </description>
</property>




-- 


*Sisu Xi, PhD Candidate*

http://www.cse.wustl.edu/~xis/
Department of Computer Science and Engineering
Campus Box 1045
Washington University in St. Louis
One Brookings Drive
St. Louis, MO 63130

--089e0149c1def5375404fe0ce05a
Content-Type: text/html; charset=UTF-8
Content-Transfer-Encoding: quoted-printable

<div dir=3D"ltr">Hi, all:<div><br></div><div>I am new to hadoop. I followed=
 the tutorial on=C2=A0</div><div><a href=3D"http://www.michael-noll.com/tut=
orials/running-hadoop-on-ubuntu-linux-multi-node-cluster/">http://www.micha=
el-noll.com/tutorials/running-hadoop-on-ubuntu-linux-multi-node-cluster/</a=
></div>

<div><br></div><div>and installed hadoop 2.2.0 on two 4-core Ubuntu 12.04 m=
achines.</div><div><br></div><div>I can start the pi program, however, only=
 the master node is doing the work (I checked the top on each machine).</di=
v>

<div>Seems the two nodes are configured correctly, because I can start =C2=
=A0the program in the slave node as well, and still only the master node is=
 doing the actual work.</div><div>I have tried different number of mappers =
for the pi program, and the results is the same.</div>

<div><br></div><div>Is there anything else I can check?=C2=A0</div><div><br=
></div><div>In the end is my configure file on each host.</div><div><br></d=
iv><div>Thanks very much!</div><div><br></div><div>Sisu</div><div><br></div=
>

<div>---------yarn-site.xml-------</div><div><div><br></div><div>&lt;proper=
ty&gt;</div><div>=C2=A0 &lt;name&gt;yarn.nodemanager.aux-services&lt;/name&=
gt;</div><div>=C2=A0 &lt;value&gt;mapreduce_shuffle&lt;/value&gt;</div><div=
>&lt;/property&gt;</div>

<div><br></div><div><br></div><div>&lt;property&gt;</div><div>=C2=A0 &lt;na=
me&gt;yarn.nodemanager.aux-services.mapreduce.shuffle.class&lt;/name&gt;</d=
iv><div>=C2=A0 &lt;value&gt;org.apache.hadoop.mapred.ShuffleHandler&lt;/val=
ue&gt;</div>

<div>&lt;/property&gt;</div><div><br></div><div><br></div><div>&lt;property=
&gt;</div><div>=C2=A0 &lt;name&gt;yarn.resourcemanager.address&lt;/name&gt;=
</div><div>=C2=A0 &lt;value&gt;master:8032&lt;/value&gt;</div><div>&lt;/pro=
perty&gt;</div>

</div><div><br></div><div>---------------hdfs-site.xml--------------------<=
/div><div><div><br></div><div>&lt;property&gt;</div><div>=C2=A0 &lt;name&gt=
;dfs.replication&lt;/name&gt;</div><div>=C2=A0 &lt;value&gt;2&lt;/value&gt;=
</div>

<div>&lt;/property&gt;</div><div><br></div><div><br></div><div>&lt;property=
&gt;</div><div>=C2=A0 &lt;name&gt;dfs.namenode.name.dir&lt;/name&gt;</div><=
div>=C2=A0 &lt;value&gt;file:/home/xisisu/mydata/hdfs/namenode&lt;/value&gt=
;</div>

<div>&lt;/property&gt;</div><div><br></div><div><br></div><div>&lt;property=
&gt;</div><div>=C2=A0 &lt;name&gt;dfs.datanode.data.dir&lt;/name&gt;</div><=
div>=C2=A0 &lt;value&gt;file:/home/xisisu/mydata/hdfs/datanode&lt;/value&gt=
;</div>

<div>&lt;/property&gt;</div></div><div><br></div><div><br></div><div>------=
-------core-site.xml-------------</div><div><br></div><div><div>&lt;propert=
y&gt;<br></div><div>&lt;name&gt;<a href=3D"http://fs.default.name">fs.defau=
lt.name</a>&lt;/name&gt;</div>

<div>&lt;value&gt;hdfs://master:9000&lt;/value&gt;</div><div>&lt;/property&=
gt;</div></div><div><br></div><div>------------------mapred-site.xml-------=
----------</div><div><div><br></div><div>&lt;property&gt;</div><div>=C2=A0 =
&lt;name&gt;<a href=3D"http://mapreduce.framework.name">mapreduce.framework=
.name</a>&lt;/name&gt;</div>

<div>=C2=A0 &lt;value&gt;yarn&lt;/value&gt;</div><div>&lt;/property&gt;</di=
v><div><br></div><div>&lt;property&gt;</div><div>=C2=A0 &lt;name&gt;mapred.=
job.tracker&lt;/name&gt;</div><div>=C2=A0 &lt;value&gt;master:54311&lt;/val=
ue&gt;</div>

<div>=C2=A0 &lt;description&gt;The host and port that the MapReduce job tra=
cker runs</div><div>=C2=A0 at. =C2=A0If &quot;local&quot;, then jobs are ru=
n in-process as a single map</div><div>=C2=A0 and reduce task.</div><div>=
=C2=A0 &lt;/description&gt;</div>

<div>&lt;/property&gt;</div></div><div><br></div><div><br></div><div><br cl=
ear=3D"all"><div><br></div>-- <br><div dir=3D"ltr"><div><img src=3D"https:/=
/dl.dropboxusercontent.com/u/268069/Sisu_Contact.jpeg" width=3D"96" height=
=3D"96"><br>

</div><div><br></div><div><b>Sisu Xi, PhD Candidate</b><br></div><br><a hre=
f=3D"http://www.cse.wustl.edu/~xis/" target=3D"_blank">http://www.cse.wustl=
.edu/~xis/</a><br>Department of Computer Science and Engineering<br>Campus =
Box 1045<br>

Washington University in St. Louis<br>One Brookings Drive<br>St. Louis, MO =
63130</div>
</div></div>

--089e0149c1def5375404fe0ce05a--