Return-Path: Delivered-To: apmail-hadoop-hdfs-user-archive@minotaur.apache.org Received: (qmail 93611 invoked from network); 2 Apr 2010 04:50:17 -0000 Received: from unknown (HELO mail.apache.org) (140.211.11.3) by 140.211.11.9 with SMTP; 2 Apr 2010 04:50:17 -0000 Received: (qmail 68770 invoked by uid 500); 2 Apr 2010 04:50:17 -0000 Delivered-To: apmail-hadoop-hdfs-user-archive@hadoop.apache.org Received: (qmail 68603 invoked by uid 500); 2 Apr 2010 04:50:17 -0000 Mailing-List: contact hdfs-user-help@hadoop.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: hdfs-user@hadoop.apache.org Delivered-To: mailing list hdfs-user@hadoop.apache.org Received: (qmail 68595 invoked by uid 99); 2 Apr 2010 04:50:16 -0000 Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 02 Apr 2010 04:50:16 +0000 X-ASF-Spam-Status: No, hits=2.2 required=10.0 tests=FREEMAIL_FROM,HTML_MESSAGE,NORMAL_HTTP_TO_IP,RCVD_IN_DNSWL_NONE,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL,WEIRD_PORT X-Spam-Check-By: apache.org Received-SPF: pass (nike.apache.org: domain of praveensripati@gmail.com designates 209.85.223.203 as permitted sender) Received: from [209.85.223.203] (HELO mail-iw0-f203.google.com) (209.85.223.203) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 02 Apr 2010 04:50:09 +0000 Received: by iwn41 with SMTP id 41so1296650iwn.20 for ; Thu, 01 Apr 2010 21:49:48 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:mime-version:received:in-reply-to:references :date:received:message-id:subject:from:to:content-type; bh=ECewVtUbYouJt6pMmV8kKyUqhq/jfIJj0e9IbhzV5wk=; b=MAxosdiaP0N1gNQlQQTKNg1CSsPNzRV/DL41Yk5eJ7xveWt8EcN64JZ9K9zruUUcp3 T8WjJCUKQ9FNRUdgsJ5RxoNZNATkFcpb2x06B5zLI4U9D5te2rigK69FAqoRC/WBeL2f QCOiiUf4pz11Ximu9EQ32G8pwV0/G1/3jvtVk= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :content-type; b=egKvxHSn3RW07yH263arsUVVy5H9mrIxh7pkMwk2rIPUQfzjc6ApDDs9NYPinBmOH7 imtVtmUOvHgtNupWxZ5m6g65VKJuULiqNnxrXIBZVupyKaZSfJd062k7j+8MNeAVIEoC P7ZuEEarOvobN0eThAmMdiVZoPNy8FKQhObic= MIME-Version: 1.0 Received: by 10.231.183.137 with HTTP; Thu, 1 Apr 2010 21:49:48 -0700 (PDT) In-Reply-To: <000301cad20a$fb0c3e70$f124bb50$@kddilabs.jp> References: <000301cad20a$fb0c3e70$f124bb50$@kddilabs.jp> Date: Fri, 2 Apr 2010 10:19:48 +0530 Received: by 10.231.190.204 with SMTP id dj12mr612646ibb.9.1270183788282; Thu, 01 Apr 2010 21:49:48 -0700 (PDT) Message-ID: Subject: Re: DataNode not able to talk to NameNode From: Praveen Sripati To: hdfs-user@hadoop.apache.org Content-Type: multipart/alternative; boundary=0016363b888c8004f3048339b557 X-Virus-Checked: Checked by ClamAV on apache.org --0016363b888c8004f3048339b557 Content-Type: text/plain; charset=ISO-8859-1 Hi, Thanks for the quick response. Wish the documentation was a bit more clear on it. Now it works. I get "Live Datanodes : 1" in the NameNode console. >> modify fs.default.name from hdfs:/localhost:9050 to hdfs://master:9050 Just curious, how does it impact the Socket on the NameNode? Praveen On Fri, Apr 2, 2010 at 7:49 AM, zhu weimin wrote: > Hi > > > > >Why is that the DataNode not able to Connect at port 9050 on the NameNode, > while the SocketClient.java connects to ?>SocketServer.java on port 9050? Is > there anything different that the NameNode creates a socket? > > > > modify fs.default.name from hdfs:/localhost:9050 to hdfs://master:9050 > > and modify mapred.job.tracker from localhost:9001 to master:9001 > > > > > > zhuweimin > > > > > > *From:* Praveen Sripati [mailto:praveensripati@gmail.com] > *Sent:* Friday, April 02, 2010 11:00 AM > *To:* hdfs-user@hadoop.apache.org > *Subject:* DataNode not able to talk to NameNode > > > > Hi, > > I am trying to setup Hadoop on a two node cluster, both using Ubuntu 9.10. > I have configured one node as NameNode/JobTracker and the other as > DataNode/TaskTracker. > > I have the following in the hosts file for the master and the slave > > master> cat /etc/hosts > 192.168.0.100 master > 192.168.0.102 slave > 127.0.0.1 localhost > > slave> cat /etc/hosts > 192.168.0.100 master > 192.168.0.102 slave > 127.0.0.1 localhosts > > and the configuration file on the master and the slave has > > master -> core-site.xml -> fs.default.name->hdfs://localhost:9050 > -> hdfs-site.xml -> dfs.replication->1 > -> mapred-site.xml -> mapred.job.tracker->localhost:9001 > > slave -> core-site.xml -> fs.default.name->hdfs://master:9050 > -> hdfs-site.xml -> dfs.replication->1 > -> mapred-site.xml -> mapred.job.tracker->master:9001 > > > When I run the command start-dfs.sh, the NameNode starts without any errors > and the script tries to start the DataNode. But, the DataNode is not able to > connect to the MasterNode. The following is in the > hadoop-praveensripati-datanode-slave.log file > > 2010-04-02 06:54:35,630 INFO org.apache.hadoop.ipc.Client: Retrying connect > to server: master/192.168.0.100:9050. Already tried 9 time(s). > 2010-04-02 06:54:35,645 INFO org.apache.hadoop.ipc.RPC: Server at master/ > 192.168.0.100:9050 not available yet, Zzzzz... > > 1. Able to ping the master from the slave and the other way. > 2. Able to ssh into slave from master and other way. > 3. Disabled ipv6 on master and slave. /etc/sysctl.conf has > net.ipv6.conf.all.disable_ipv6 = 1. > > I wrote a Java SocketClient Program to connect from the DataNode to the > NameNode at port 9050 and I get the following exception > > java.net.ConnectException: Connection refused > at java.net.PlainSocketImpl.socketConnect(Native Method) > at java.net.PlainSocketImpl.doConnect(PlainSocketImpl.java:333) > at java.net.PlainSocketImpl.connectToAddress(PlainSocketImpl.java:195) > at java.net.PlainSocketImpl.connect(PlainSocketImpl.java:182) > at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:366) > at java.net.Socket.connect(Socket.java:525) > at SocketClient.main(SocketClient.java:23) > > Then, I stop the NameNode and DataNode and then by using Java Programs I > create a socket (at 9050) on the NameNode and am able to connect from the > DataNode using Java Program. > > ServerSocket.java has > > int port = Integer.parseInt(args[0]); > ServerSocket srv = new ServerSocket(port); > Socket socket = srv.accept(); > > SocketClient.java has > > InetAddress addr = InetAddress.getByName(args[0]); > int port = Integer.parseInt(args[1]); > SocketAddress sockaddr = new InetSocketAddress(addr, port); > Socket sock = new Socket(); > int timeoutMs = 2000; > sock.connect(sockaddr, timeoutMs); > > When I do 'netstat -a | grep 9050' I get > > When NameNode creates the Socket -> tcp 0 0 > localhost:9050 *:* LISTEN > When Java Program creates a Socket -> tcp 0 0 > *:9050 *:* LISTEN > > Why is that the DataNode not able to Connect at port 9050 on the NameNode, > while the SocketClient.java connects to SocketServer.java on port 9050? Is > there anything different that the NameNode creates a socket? > -- > Praveen > -- Praveen --0016363b888c8004f3048339b557 Content-Type: text/html; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable
Hi,

= Thanks for the quick response. Wish the documentation was a bit more clear = on it.

Now it works. I get "Live Datanodes : 1" in the NameNode consol= e.

>> modify =A0fs.def= ault.name from=A0 hdfs:/localhost:9050 =A0to hdfs://master:9050

Just curious, how does it impact the Socket on the = NameNode?

Praveen

On Fri, Apr 2, 2010 at 7:49 AM, zhu w= eimin <xim-= shu@tsm.kddilabs.jp> wrote:

Hi

=A0

>Why is that the DataNode not able to Connect at port 9050 on the NameNode, while the SocketClient.java connects = to ?>SocketServer.java on port 9050? Is there anything different that the NameNode creates a socke= t?

=A0

modify =A0fs.default.name from=A0 hdfs:/localhost:9050 =A0to hdfs://master:9050

=A0and modify mapred.job.tracker from localhost:9001 to master:9001

=A0

=A0

zhuweimin

=A0

=A0

From: Praveen Sripati [mailto:prave= ensripati@gmail.com]
Sent: Friday, April 02, 2010 11:00 AM
To: hdfs-user@hadoop.apache.org
Subject: DataNode not able to talk to NameNode

=A0

Hi,

I am trying to setup Hadoop on a two node cluster, both using Ubuntu 9.10. = I have configured one node as NameNode/JobTracker and the other as DataNode/TaskTracker.

I have the following in the hosts file for the master and the slave

master> cat /etc/hosts
192.168.0.100 master
192.168.0.102 slave
127.0.0.1 localhost

slave> cat /etc/hosts
192.168.0.100 master
192.168.0.102 slave
127.0.0.1 localhosts

and the configuration file on the master and the slave has

master =A0=A0=A0 -> core-site.xml -> fs.default.name->hdfs://localhost:9050
=A0=A0=A0 -> hdfs-site.xml -> dfs.replication->1
=A0=A0=A0 -> mapred-site.xml -> mapred.job.tracker->localhost:9001

slave =A0=A0=A0 -> core-site.xml -> fs.default.name->hdfs://master:9050
=A0=A0=A0 -> hdfs-site.xml -> dfs.replication->1
=A0=A0=A0 -> mapred-site.xml -> mapred.job.tracker->master:9001


When I run the command start-dfs.sh, the NameNode starts without any errors= and the script tries to start the DataNode. But, the DataNode is not able to connect to the MasterNode. The following is in the hadoop-praveensripati-datanode-slave.log file

2010-04-02 06:54:35,630 INFO org.apache.hadoop.ipc.Client: Retrying connect= to server: master/192.= 168.0.100:9050. Already tried 9 time(s).
2010-04-02 06:54:35,645 INFO org.apache.hadoop.ipc.RPC: Server at master/192.168.0.100:9050 not available yet, Zzzzz...

1. Able to ping the master from the slave and the other way.
2. Able to ssh into slave from master and other way.
3. Disabled ipv6 on master and slave. /etc/sysctl.conf has net.ipv6.conf.all.disable_ipv6 =3D 1.

I wrote a Java SocketClient Program to connect from the DataNode to the NameNode at port 9050 and I get the following exception

java.net.ConnectException: Connection refused
=A0=A0=A0 at java.net.PlainSocketImpl.socketConnect(Native Method)
=A0=A0=A0 at java.net.PlainSocketImpl.doConnect(PlainSocketImpl.java:333) =A0=A0=A0 at java.net.PlainSocketImpl.connectToAddress(PlainSocketImpl.java:195)
=A0=A0=A0 at java.net.PlainSocketImpl.connect(PlainSocketImpl.java:182)
=A0=A0=A0 at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:366)
=A0=A0=A0 at java.net.Socket.connect(Socket.java:525)
=A0=A0=A0 at SocketClient.main(SocketClient.java:23)

Then, I stop the NameNode and DataNode and then by using Java Programs I cr= eate a socket (at 9050) on the NameNode and am able to connect from the DataNode using Java Program.

ServerSocket.java has

=A0=A0=A0 int port =3D Integer.parseInt(args[0]);
=A0=A0=A0 ServerSocket srv =3D new ServerSocket(port);
=A0=A0=A0 Socket socket =3D srv.accept();

SocketClient.java has

=A0=A0=A0 InetAddress addr =3D InetAddress.getByName(args[0]);
=A0=A0=A0 int port =3D Integer.parseInt(args[1]);
=A0=A0=A0 SocketAddress sockaddr =3D new InetSocketAddress(addr, port);
=A0=A0=A0 Socket sock =3D new Socket();
=A0=A0=A0 int timeoutMs =3D 2000;
=A0=A0=A0 sock.connect(sockaddr, timeoutMs);

When I do 'netstat -a | grep 9050' I get

When NameNode creates the Socket =A0=A0=A0 -> tcp=A0=A0=A0=A0=A0=A0=A0 0=A0=A0=A0=A0=A0 0 localhost:9050=A0=A0=A0=A0=A0=A0=A0=A0=A0 *:*=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0 LISTEN
When Java Program creates a Socket =A0=A0=A0 -> tcp=A0=A0=A0=A0=A0=A0=A0 0=A0=A0=A0=A0=A0 0 *:9050=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0 *:*=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0 LISTEN

Why is that the DataNode not able to Connect at port 9050 on the NameNode, while the SocketClient.java connects to SocketServer.java on port 9050? Is there anything different that the NameNode creates a socket?
--
Praveen




--
Praveen
--0016363b888c8004f3048339b557--