hadoop-hdfs-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Marcin Mejran <marcin.mej...@hooklogic.com>
Subject RE: Namenode formatting problem
Date Wed, 20 Feb 2013 01:46:08 GMT
The issue may be that the nodes are trying to use the ec2 public ip (which would be used for
external access) to access each other which does not work (or doesn't work trivially). You
need to use the private ips which are given by ifconfig.

ec2 gives you static ips as long as you don't restart or stop/start an instance.

That said, it gives you TWO ips and you need to be careful on which you use:
* Private ip: This is a local ip that cannot be access from outside ec2 but can be used to
communicate between instance. This is what ifconfig returns.
* Public ip: This ip can be used for external access but it's not shown in ifconfig.
You can imagine each instance as having its own personal NAT.

Instances should use the private ip when communicating to each other. I'm not sure if the
public ip cannot be used or is just a giant pain to setup correctly. I just did a test on
my company ec2 instances and I can ping the public ip from an instance but I cannot use it
for ssh. Not sure why off hand although I believe any data sent to a public ip (even from
within ec2) gets charged transfer fees so it's not a good idea in any regard.


-----Original Message-----
From: Harsh J [mailto:harsh@cloudera.com] 
Sent: Tuesday, February 19, 2013 11:25 AM
To: <user@hadoop.apache.org>
Subject: Re: Namenode formatting problem

To simplify my previous post, your IPs for the master/slave/etc. in /etc/hosts file should
match the ones reported by "ifconfig" always.
In proper deployments, IP is static. If IP is dynamic, we'll need to think of some different

On Tue, Feb 19, 2013 at 9:53 PM, Harsh J <harsh@cloudera.com> wrote:
> Hey Keith,
> I'm guessing whatever "ip-13-0-177-110" is resolving to (ping to 
> check), is not what is your local IP on that machine (or rather, it 
> isn't the machine you intended to start it on)?
> Not sure if EC2 grants static IPs, but otherwise a change in the 
> assigned IP (checkable via ifconfig) would probably explain the 
> "Cannot assign" error received when we tried a bind() syscall.
> On Tue, Feb 19, 2013 at 4:30 AM, Keith Wiley <kwiley@keithwiley.com> wrote:
>> This is Hadoop 2.0.  Formatting the namenode produces no errors in the shell, but
the log shows this:
>> 2013-02-18 22:19:46,961 FATAL 
>> org.apache.hadoop.hdfs.server.namenode.NameNode: Exception in 
>> namenode join
>> java.net.BindException: Problem binding to [ip-13-0-177-110:9212] java.net.BindException:
Cannot assign requested address; For more details see:  http://wiki.apache.org/hadoop/BindException
>>         at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:710)
>>         at org.apache.hadoop.ipc.Server.bind(Server.java:356)
>>         at org.apache.hadoop.ipc.Server$Listener.<init>(Server.java:454)
>>         at org.apache.hadoop.ipc.Server.<init>(Server.java:1833)
>>         at org.apache.hadoop.ipc.RPC$Server.<init>(RPC.java:866)
>>         at org.apache.hadoop.ipc.ProtobufRpcEngine$Server.<init>(ProtobufRpcEngine.java:375)
>>         at org.apache.hadoop.ipc.ProtobufRpcEngine.getServer(ProtobufRpcEngine.java:350)
>>         at org.apache.hadoop.ipc.RPC.getServer(RPC.java:695)
>>         at org.apache.hadoop.ipc.RPC.getServer(RPC.java:684)
>>         at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.<init>(NameNodeRpcServer.java:238)
>>         at org.apache.hadoop.hdfs.server.namenode.NameNode.createRpcServer(NameNode.java:452)
>>         at org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:434)
>>         at org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:608)
>>         at org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:589)
>>         at org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1140)
>>         at 
>> org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:12
>> 04)
>> 2013-02-18 22:19:46,988 INFO org.apache.hadoop.util.ExitUtil: Exiting 
>> with status 1
>> 2013-02-18 22:19:46,990 INFO org.apache.hadoop.hdfs.server.namenode.NameNode: SHUTDOWN_MSG:
>> /************************************************************
>> SHUTDOWN_MSG: Shutting down NameNode at ip-13-0-177-11/ 
>> ************************************************************/
>> No java processes begin (although I wouldn't expect formatting the namenode to start
any processes, only starting the namenode or datanode should do that), and "hadoop fs -ls
/" gives me this:
>> ls: Call From [CLIENT_HOST]/ to [MASTER_HOST]:9000 failed on 
>> connection exception: java.net.ConnectException: Connection refused; 
>> For more details see:  
>> http://wiki.apache.org/hadoop/ConnectionRefused
>> My /etc/hosts looks like this:
>>   localhost localhost.localdomain CLIENT_HOST
>> This is on EC2.  All of the nodes are in the same security group and the security
group has full inbound access.  I can ssh between all three machines (client/master/slave)
without a password ala authorized_keys.  I can ping the master node from the client machine
(although I don't know how to ping a specific port, such as the hdfs port (9000)).  Telnet
doesn't behave on EC2 which makes port testing a little difficult.
>> Any ideas?
>> ________________________________________________________________________________
>> Keith Wiley     kwiley@keithwiley.com     keithwiley.com    music.keithwiley.com
>> "The easy confidence with which I know another man's religion is 
>> folly teaches me to suspect that my own is also."
>>                                            --  Mark Twain 
>> _____________________________________________________________________
>> ___________
> --
> Harsh J

Harsh J

View raw message