Return-Path: X-Original-To: apmail-hadoop-user-archive@minotaur.apache.org Delivered-To: apmail-hadoop-user-archive@minotaur.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 325F717ACE for ; Wed, 23 Sep 2015 20:30:27 +0000 (UTC) Received: (qmail 59242 invoked by uid 500); 23 Sep 2015 20:30:13 -0000 Delivered-To: apmail-hadoop-user-archive@hadoop.apache.org Received: (qmail 59134 invoked by uid 500); 23 Sep 2015 20:30:13 -0000 Mailing-List: contact user-help@hadoop.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@hadoop.apache.org Delivered-To: mailing list user@hadoop.apache.org Received: (qmail 59124 invoked by uid 99); 23 Sep 2015 20:30:13 -0000 Received: from Unknown (HELO spamd1-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 23 Sep 2015 20:30:13 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd1-us-west.apache.org (ASF Mail Server at spamd1-us-west.apache.org) with ESMTP id E3EB8F775B for ; Wed, 23 Sep 2015 20:30:12 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd1-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: 2.903 X-Spam-Level: ** X-Spam-Status: No, score=2.903 tagged_above=-999 required=6.31 tests=[DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, HTML_MESSAGE=3, NORMAL_HTTP_TO_IP=0.001, URIBL_BLOCKED=0.001, WEIRD_PORT=0.001] autolearn=disabled Authentication-Results: spamd1-us-west.apache.org (amavisd-new); dkim=pass (2048-bit key) header.d=gmail.com Received: from mx1-us-west.apache.org ([10.40.0.8]) by localhost (spamd1-us-west.apache.org [10.40.0.7]) (amavisd-new, port 10024) with ESMTP id a2eIE1zgCzMz for ; Wed, 23 Sep 2015 20:29:59 +0000 (UTC) Received: from mail-yk0-f177.google.com (mail-yk0-f177.google.com [209.85.160.177]) by mx1-us-west.apache.org (ASF Mail Server at mx1-us-west.apache.org) with ESMTPS id 346A920F4A for ; Wed, 23 Sep 2015 20:29:59 +0000 (UTC) Received: by ykft14 with SMTP id t14so53981215ykf.0 for ; Wed, 23 Sep 2015 13:29:52 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :content-type; bh=Kx7+uqVj2QkPj+Q285ShJYhpwzn7/5vffEgg0wHHFpk=; b=TSW/TZI8mojLIazZYCP3UMrY5Z5J+xdDgHWouuosFJsQUezj51O5FAw0FgTAwVwlnD EgtENilR/Dq6Gmrj8sEm/OQ7lIBKrT84nuKH8qf8u8Ol66bdXF4tV6POI725f0KQoC/B zQW7jX67V4MPJ+wuvDrfUCCaVNKCEHdWhCZsnpMSoOlH4/1Rs+UF7OArCfrP5cISuLWd Ok/CAlZyimUwtkCA3Ey1nPEBRCThMslxydDfXC/w22HVwY33ETokr8QfgmTC1iDD5Gks VeyhOgfh7XrVOoRO+2BcsUSQ1b218Bc6Xk1qZ3PTSdgajOD6b82AE4JyTZO3c/ssmyuM 7iwQ== MIME-Version: 1.0 X-Received: by 10.170.164.9 with SMTP id g9mr28072664ykd.49.1443040191983; Wed, 23 Sep 2015 13:29:51 -0700 (PDT) Received: by 10.37.210.195 with HTTP; Wed, 23 Sep 2015 13:29:51 -0700 (PDT) In-Reply-To: References: Date: Wed, 23 Sep 2015 15:29:51 -0500 Message-ID: Subject: Re: Help troubleshooting multi-cluster setup From: Daniel Watrous To: user@hadoop.apache.org Content-Type: multipart/alternative; boundary=001a113a6cda0a398a05206ffad6 --001a113a6cda0a398a05206ffad6 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable I was able to get the jobs submitting to the cluster by adding the following property to mapred-site.xml mapreduce.framework.name yarn I also had to add the following properties to yarn-site.xml yarn.nodemanager.aux-services mapreduce_shuffle yarn.nodemanager.aux-services.mapreduce_shuffle.class org.apache.hadoop.mapred.ShuffleHandler I'm still not sure why the datanodes don't show up in the nodes view. Is the idea that a data node is only used for HDFS and yarn doesn't schedule jobs there? If so, how can I add additional compute hosts? What are those called? On Wed, Sep 23, 2015 at 3:08 PM, Daniel Watrous wrote: > I'm not sure if this is related, but I'm seeing some errors > in hadoop-hadoop-namenode-hadoop-master.log > > 2015-09-23 19:56:27,798 WARN org.apache.hadoop.hdfs.server.blockmanagemen= t.DatanodeManager: Unresolved datanode registration: hostname cannot be res= olved (ip=3D192.168.51.1, hostname=3D192.168.51.1) > 2015-09-23 19:56:27,800 INFO org.apache.hadoop.ipc.Server: IPC Server han= dler 6 on 54310, call org.apache.hadoop.hdfs.server.protocol.DatanodeProtoc= ol.registerDatanode from 192.168.51.1:54554 Call#373 Retry#0 > org.apache.hadoop.hdfs.server.protocol.DisallowedDatanodeException: Datan= ode denied communication with namenode because hostname cannot be resolved = (ip=3D192.168.51.1, hostname=3D192.168.51.1): DatanodeRegistration(0.0.0.0:= 50010, datanodeUuid=3D8a5d90c8-b909-46d3-80ec-2a3a8f1fe904, infoPort=3D5007= 5, infoSecurePort=3D0, ipcPort=3D50020, storageInfo=3Dlv=3D-56;cid=3DCID-bc= 60d031-11b0-4eb5-8f9b-da0f8a069ea6;nsid=3D1223814533;c=3D0) > at org.apache.hadoop.hdfs.server.blockmanagement.DatanodeManager.registe= rDatanode(DatanodeManager.java:863) > at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.registerDatanode(= FSNamesystem.java:4529) > at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.registerData= node(NameNodeRpcServer.java:1279) > at org.apache.hadoop.hdfs.protocolPB.DatanodeProtocolServerSideTranslato= rPB.registerDatanode(DatanodeProtocolServerSideTranslatorPB.java:95) > at org.apache.hadoop.hdfs.protocol.proto.DatanodeProtocolProtos$Datanode= ProtocolService$2.callBlockingMethod(DatanodeProtocolProtos.java:28539) > at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.cal= l(ProtobufRpcEngine.java:616) > at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:969) > at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2049) > at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2045) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:422) > at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInforma= tion.java:1657) > at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2043) > > > I don't have a server with the IP 192.168.51.1 and I don't think I'm > referencing that anywhere. Is there some reason that it's trying to add > that host as a namenode? > > On Wed, Sep 23, 2015 at 1:58 PM, Daniel Watrous > wrote: > >> Hi, >> >> I have deployed a multi-node cluster with one master and two data nodes. >> Here's what jps shows: >> >> hadoop@hadoop-master:~$ jps >> 24641 SecondaryNameNode >> 24435 DataNode >> 24261 NameNode >> 24791 ResourceManager >> 25483 Jps >> 24940 NodeManager >> >> hadoop@hadoop-data1:~$ jps >> 15556 DataNode >> 16198 NodeManager >> 16399 Jps >> >> hadoop@hadoop-data2:~$ jps >> 16418 Jps >> 15575 DataNode >> 16216 NodeManager >> >> When I open the web console, I only see one node running: >> http://screencast.com/t/E6yehRvUbt >> >> Where are the other two nodes? Why don't they show up? >> >> Next I run one of the example scripts >> >> hadoop@hadoop-master:~$ hadoop jar >> /usr/local/hadoop/share/hadoop/mapreduce/hadoop-mapreduce-examples-2.7.1= .jar >> pi 10 30 >> Number of Maps =3D 10 >> Samples per Map =3D 30 >> Wrote input for Map #0 >> Wrote input for Map #1 >> ... >> Job Finished in 2.956 seconds >> Estimated value of Pi is 3.14146666666666666667 >> >> I can't see this anywhere in the web interface. I thought it might show >> in the Applications sub-menu. Should I be able to see this? It appears t= o >> run successfully. >> >> Daniel >> > > --001a113a6cda0a398a05206ffad6 Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: quoted-printable
I was able to get the jobs submitting to the cluster by ad= ding the following property to mapred-site.xml

=C2=A0 = =C2=A0<property>=C2=A0=C2=A0
=C2=A0=C2=A0=C2=A0=C2=A0<name>mapreduce.framework.name</name>=C2=A0=C2=A0
=C2=A0=C2=A0=C2=A0=C2=A0<value>yarn</value>=C2=A0=C2=A0
=C2=A0=C2=A0</property>

I also had to ad= d the following properties to yarn-site.xml

=C2=A0= =C2=A0<property>
=C2=A0=C2=A0=C2=A0=C2=A0<name>yarn.nodemanager.aux-services</name&= gt;
=C2=A0=C2=A0=C2=A0=C2=A0<value>mapreduce_shuffle</value>
=C2=A0=C2=A0</property>
=C2=A0=C2=A0<property>
=C2=A0=C2=A0=C2=A0=C2=A0<name>yarn.nodemanager.aux-services.mapreduce= _shuffle.class</name>
=C2=A0=C2=A0=C2=A0=C2=A0<value>org.apache.hadoop.mapred.ShuffleHandle= r</value>
=C2=A0=C2=A0</property>

I'm still no= t sure why the datanodes don't show up in the nodes view. Is the idea t= hat a data node is only used for HDFS and yarn doesn't schedule jobs th= ere? If so, how can I add additional compute hosts? What are those called?<= /div>

On Wed= , Sep 23, 2015 at 3:08 PM, Daniel Watrous <dwmaillist@gmail.com>= wrote:
I'm n= ot sure if this is related, but I'm seeing some errors in=C2=A0hadoop-h= adoop-namenode-hadoop-master.log

2015-09-23 19:56:27,798 =
WARN org.apache.hadoop.hdfs.server.blockmanagement.DatanodeManager: Unresol=
ved datanode registration: hostname cannot be resolved (ip=3D192.168.51.1, =
hostname=3D192.168.51.1)
2015-09-23 19:56:27,800 INFO org.apache.hadoop.ipc.Server: IPC Server handl=
er 6 on 54310, call org.apache.hadoop.hdfs.server.protocol.DatanodeProtocol=
.registerDatanode from 192.168.51.1:54554 Call#373 Retry#0
org.apache.hadoop.hdfs.server.protocol.DisallowedDatanodeException: Datanod=
e denied communication with namenode because hostname cannot be resolved (i=
p=3D192.168.51.1, hostname=3D192.168.51.1): DatanodeRegistration(0.0.0.0:50010, datanodeUuid=3D=
8a5d90c8-b909-46d3-80ec-2a3a8f1fe904, infoPort=3D50075, infoSecurePort=3D0,=
 ipcPort=3D50020, storageInfo=3Dlv=3D-56;cid=3DCID-bc60d031-11b0-4eb5-8f9b-=
da0f8a069ea6;nsid=3D1223814533;c=3D0)
	at org.apache.hadoop.hdfs.server.blockmanagement.DatanodeManager.registerD=
atanode(DatanodeManager.java:863)
	at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.registerDatanode(FS=
Namesystem.java:4529)
	at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.registerDatano=
de(NameNodeRpcServer.java:1279)
	at org.apache.hadoop.hdfs.protocolPB.DatanodeProtocolServerSideTranslatorP=
B.registerDatanode(DatanodeProtocolServerSideTranslatorPB.java:95)
	at org.apache.hadoop.hdfs.protocol.proto.DatanodeProtocolProtos$DatanodePr=
otocolService$2.callBlockingMethod(DatanodeProtocolProtos.java:28539)
	at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(=
ProtobufRpcEngine.java:616)
	at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:969)
	at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2049)
	at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2045)
	at java.security.AccessController.doPrivileged(Native Method)
	at javax.security.auth.Subject.doAs(Subject.java:422)
	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformati=
on.java:1657)
	at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2043)

I don't have a server with the IP=C2=A0= 192.168.51.1 and I do= n't think I'm referencing that anywhere. Is there some reason that = it's trying to add that host as a namenode?

On Wed, Sep 23, 2015 at 1:58 PM, Daniel Watrous <dwmailli= st@gmail.com> wrote:
Hi,

I have deployed a multi-node cluster with= one master and two data nodes. Here's what jps shows:

hadoop@hadoop-master:~$ jps
24641 SecondaryNameNode=
24435 DataNode
24261 NameNode
24791 Resource= Manager
25483 Jps
24940 NodeManager
hadoop@hadoop-data1:~$ jps
15556 DataNode
<= div>16198 NodeManager
16399 Jps

hadoop@hadoop-data2:~$ jps
16418 Jps
15575 DataNode<= /div>
16216 NodeManager

When I open the = web console, I only see one node running:=C2=A0http://screencast.com/t/E6yehRvUbt=

Where are the other two nodes? Why don't they= show up?

Next I run one of the example scripts

hadoop@hadoop-master:~$ hadoop jar /usr/local/h= adoop/share/hadoop/mapreduce/hadoop-mapreduce-examples-2.7.1.jar pi 10 30
Number of Maps =C2=A0=3D 10
Samples per Map = =3D 30
Wrote input for Map #0
Wrote inpu= t for Map #1
...
Job Finished in 2.956 secon= ds
Estimated value of Pi is 3.14146666666666666667

I can't see this anywhere in the web interface. I thou= ght it might show in the Applications sub-menu. Should I be able to see thi= s? It appears to run successfully.
=
Daniel


--001a113a6cda0a398a05206ffad6--