Return-Path: X-Original-To: apmail-hadoop-user-archive@minotaur.apache.org Delivered-To: apmail-hadoop-user-archive@minotaur.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 1E19D11B45 for ; Sat, 16 Aug 2014 09:50:37 +0000 (UTC) Received: (qmail 47629 invoked by uid 500); 16 Aug 2014 09:50:26 -0000 Delivered-To: apmail-hadoop-user-archive@hadoop.apache.org Received: (qmail 47509 invoked by uid 500); 16 Aug 2014 09:50:26 -0000 Mailing-List: contact user-help@hadoop.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@hadoop.apache.org Delivered-To: mailing list user@hadoop.apache.org Received: (qmail 47499 invoked by uid 99); 16 Aug 2014 09:50:26 -0000 Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230) by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 16 Aug 2014 09:50:26 +0000 X-ASF-Spam-Status: No, hits=1.5 required=5.0 tests=HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (nike.apache.org: domain of yanbohappy@gmail.com designates 209.85.212.170 as permitted sender) Received: from [209.85.212.170] (HELO mail-wi0-f170.google.com) (209.85.212.170) by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 16 Aug 2014 09:49:59 +0000 Received: by mail-wi0-f170.google.com with SMTP id f8so1821081wiw.5 for ; Sat, 16 Aug 2014 02:49:58 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :content-type; bh=zbbJH0sLFvaucSaGBuy4ZFO0y+Dv+dyEPgqMeSZMSaQ=; b=pJR8jsrgcwZQ4el93nJR0uCXSx62HQZVEerudhI/mrXW2RyEUDdVFX1tG0EJGaC0ba a4E3rW/4dNByvnjqG8MbxmZUwz6A+cCkB8wX5Tjb34WOYZW4gqDBEYv73ct9w1wVyPdt TavWVyKwViwJTow8L7LJZtbSHuEf3t7fu1bR+oqYAHUWMFNq4PKyNvOkFwzzT2baHP7D Ch4q6O4yTMCpmjqIpH+Stol/1u7FcXwGHNKURok1wn3lI5s2MLAfR9FH/NKN/J3DEo1X STEviybdzKdiWSJEOaughRnmfvLaFqoFf5xa/6lEgdaXw05znzvw2suSAX+6gfVdUkhh +fBA== MIME-Version: 1.0 X-Received: by 10.180.93.8 with SMTP id cq8mr26694771wib.17.1408182598549; Sat, 16 Aug 2014 02:49:58 -0700 (PDT) Received: by 10.217.56.1 with HTTP; Sat, 16 Aug 2014 02:49:58 -0700 (PDT) In-Reply-To: <0ADC43F07801E94E912754005746A4E04FB46AA9@DFWX10HMPTC01.AMER.DELL.COM> References: <0ADC43F07801E94E912754005746A4E04FB46AA9@DFWX10HMPTC01.AMER.DELL.COM> Date: Sat, 16 Aug 2014 17:49:58 +0800 Message-ID: Subject: Re: Problems with the Fedarated name node configuration From: Yanbo Liang To: "user@hadoop.apache.org" Content-Type: multipart/alternative; boundary=f46d043892b190c7550500bc0f5f X-Virus-Checked: Checked by ClamAV on apache.org --f46d043892b190c7550500bc0f5f Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable - Do you see anything wrong in above configuration ? Looks like all right. - Where am I supposed to run this ( on name nodes, data nodes or on every node) ? run on all DataNodes, refresh all DataNodes to pick up the newly added NameNode. - I suppose the default data node rpc port is =E2=80=9C8020=E2=80= =9D =E2=80=93 and I should be able to set it by a property in hdfs-site.xml ( dfs.datanode.ipc.address ) =E2=80=93 is that correct ? Yes. - Regarding SSH configuration =E2=80=93 I have created a ssh cert = only on my primary node ( using ssh-keygen) and copied it on all other data and the new name nodes using ssh-copy-id. Would it be necessary to create cert for the new name node as well ? Yes. 2014-08-16 13:35 GMT+08:00 : > Hello, > > > > I am a HDFS newbie > > I am using Hadoop version 2.4.1 > > And following instructions for cluster set-up from > > > http://hadoop.apache.org/docs/r2.4.1/hadoop-project-dist/hadoop-common/Cl= usterSetup.html > > and for namenode federation from > > > http://hadoop.apache.org/docs/r2.3.0/hadoop-project-dist/hadoop-hdfs/Fede= ration.html > > > > I have set-up a HDFS cluster with one name-node and two data-nodes > successfully (with ease J) > > > > I however am having challenges setting up a federated name-node. > > > > All my machines are Suse Linux SLES 11 > > > > Here are the steps that I followed for adding a federated name node to my > working cluster > > > > Step1: I set up a new SLES11 VM and installed HDFS on that > > > > Step2 : Changed config in my hdfs-site.xml as follows and deployed on all > machines > > > > > > > > > > dfs.nameservices > > ns1,ns2 > > > > dfs.namenode.name.dir > > file:/home/hduser/mydata/hdfs/namenode > > Space for name node to persist stuff > > > > > > dfs.namenode.rpc-address.ns1 > > sles-hdfs1:9000 > > > > > > dfs.namenode.http-address.ns1 > > sles-hdfs1:50070 > > > > > > dfs.namenode.rpc-address.ns2 > > sles-hdfs4:9000 > > > > > > dfs.namenode.http-address.ns2 > > sles-hdfs2:50070 > > > > > > dfs.namenode.hosts > > sles-hdfs2,sles-hdfs5 > > List of allowed data nodes > > > > > > > > Step3: I formatted my new name with same cluster id that I used for my > first (working) name node > > hdfs namenode -format -clusterId CID-085f6f5f-784f-4b00-b3bf-937f2dd7808= a > > > > Step4: I start the new name node and it starts successfully > > hadoop-daemon.sh --config $HADOOP_CONF_DIR --script hdfs start namenode > > > > Hereafter =E2=80=93 The instructions are somewhat unclear =E2=80=93 > > > > Next I am supposed to run the command > > > > $ hdfs dfadmin -refreshNameNode : > > > > Questions that I have are > > - Do you see anything wrong in above configuration ? > > - Where am I supposed to run this ( on name nodes, data nodes or > on every node) ? > > - I suppose the default data node rpc port is =E2=80=9C8020=E2= =80=9D =E2=80=93 and I > should be able to set it by a property in hdfs-site.xml ( dfs.datanode.i= pc.address > ) =E2=80=93 is that correct ? > > - Regarding SSH configuration =E2=80=93 I have created a ssh cer= t only > on my primary node ( using ssh-keygen) and copied it on all other data an= d > the new name nodes using ssh-copy-id. Would it be necessary to create cer= t > for the new name node as well ? > > > > Just FYI =E2=80=93 server names of my nodes > > sles-hdfs1- primary name node > > sles-hdfs2-One Data node > > sles-hdfs5- Another Data node > > sles-hdfs4 =E2=80=93new federated name node > > > > > > > --f46d043892b190c7550500bc0f5f Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: quoted-printable

-=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0Do you see anything wrong in above= configuration ?

Looks like all right.

-=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= Where am I supposed to run this ( on name nodes, data nodes or on every nod= e) ?

run on all DataNodes, refresh all DataNodes to pick up the newly added Name= Node.

-=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0I suppose the default data node rpc port is =E2=80=9C8020=E2=80=9D =E2=80= =93 and I should be able to set it by a property in hdfs-site.xml ( =C2=A0<= code style=3D"font-family:'Courier New'">dfs.datanode.ipc.address ) =E2=80=93 is that correct ?

Yes.

-=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= Regarding SSH configuration =E2=80=93 I have created a ssh cert only on my = primary node ( using ssh-keygen) and copied it on all other data and the ne= w name nodes using ssh-copy-id. Would it be necessary to create cert for th= e new name node as well ?

Yes.


=
2014-08-16 13:35 GMT+08:00 <Yogesh_Devi= @dell.com>:

Hello,

=C2=A0

I am a HDFS newbie

I am using Hadoop version 2.4.1

And following instructions for cluster set-up from <= u>

http://h= adoop.apache.org/docs/r2.4.1/hadoop-project-dist/hadoop-common/ClusterSetup= .html

and for namenode federation from

http://hadoo= p.apache.org/docs/r2.3.0/hadoop-project-dist/hadoop-hdfs/Federation.html

=C2=A0

I have set-up a HDFS cluster with one name-node and = two data-nodes successfully (with ease J)

=C2=A0

I however am having challenges setting up a federate= d name-node.

=C2=A0

All my machines are Suse Linux SLES 11=

=C2=A0

Here are the steps that I followed for adding a fede= rated name node to my working cluster

=C2=A0

Step1: I set up a new SLES11 VM and installed HDFS o= n that

=C2=A0

Step2 : Changed config in my hdfs-site.xml as follow= s and deployed on all machines

=C2=A0

<configuration>

<property>

<property>

=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 <name&= gt;dfs.nameservices</name>

=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 =C2=A0<value= >ns1,ns2</value>

</property>

=C2=A0 <name>dfs.namenode.name.dir</name>= ;

=C2=A0 <value>file:/home/hduser/mydata/hdfs/na= menode</value>

=C2=A0 <description>Space for name node to per= sist stuff</description>

</property>

<property>

=C2=A0=C2=A0=C2=A0 <name>dfs.namenode.rpc-addr= ess.ns1</name>

=C2=A0=C2=A0=C2=A0 <value>sles-hdfs1:9000</= value>

</property>

<property>

=C2=A0=C2=A0=C2=A0 <name>dfs.namenode.http-add= ress.ns1</name>

=C2=A0=C2=A0=C2=A0 <value>sles-hdfs1:50070<= /value>

</property>

<property>

=C2=A0=C2=A0=C2=A0 <name>dfs.namenode.rpc-addr= ess.ns2</name>

=C2=A0=C2=A0=C2=A0 <value>sles-hdfs4:9000</= value>

</property>

<property>

=C2=A0=C2=A0=C2=A0 <name>dfs.namenode.http-add= ress.ns2</name>

=C2=A0=C2=A0=C2=A0 <value>sles-hdfs2:50070<= /value>

</property>

<property>

=C2=A0 <name>dfs.namenode.hosts</name>

=C2=A0 <value>sles-hdfs2,sles-hdfs5</value&= gt;

=C2=A0 <description>List of allowed data nodes= </description>

</property>

</configuration>

=C2=A0

Step3: I formatted my new name with same cluster id = that I used for my first (working) name node

hdfs namenode -format -clusterId=C2=A0 CID-085f6f5f-= 784f-4b00-b3bf-937f2dd7808a

=C2=A0

Step4: I start the new name node and it starts succe= ssfully

hadoop-daemon.sh --config $HADOOP_CONF_DIR --script = hdfs start namenode

=C2=A0

Hereafter =E2=80=93 The instructions are somewhat un= clear =E2=80=93

=C2=A0

Next I am supposed to run the command<= /p>

=C2=A0

$ hdfs dfadmin -refres= hNameNode <datanode_host_name>:<datanode_rpc_port>

=C2=A0

Questions that I have are

-=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 Do you see anything wrong in above configuration ?<= /u>

-=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 Where am I supposed to run this ( on name nodes, data = nodes or on every node) ?

-= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 I suppose the default data node rpc port= is =E2=80=9C8020=E2=80=9D =E2=80=93 and I should be able to set it by a pr= operty in hdfs-site.xml ( =C2=A0dfs.= datanode.ipc.address ) =E2=80=93 is that correct ?

-=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 Regarding SSH configuration =E2=80=93 I have created a= ssh cert only on my primary node ( using ssh-keygen) and copied it on all = other data and the new name nodes using ssh-copy-id. Would it be necessary = to create cert for the new name node as well ?

=C2=A0

Just FYI =E2=80=93 server names of my nodes

sles-hdfs1- primary name node

sles-hdfs2-One Data node

sles-hdfs5- Another Data node

sles-hdfs4 =E2=80=93new federated name node <= u>

=C2=A0

=C2=A0

=C2=A0


--f46d043892b190c7550500bc0f5f--