Return-Path: X-Original-To: apmail-hadoop-hdfs-user-archive@minotaur.apache.org Delivered-To: apmail-hadoop-hdfs-user-archive@minotaur.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 7274D102F9 for ; Tue, 13 Aug 2013 15:58:50 +0000 (UTC) Received: (qmail 16044 invoked by uid 500); 13 Aug 2013 15:58:45 -0000 Delivered-To: apmail-hadoop-hdfs-user-archive@hadoop.apache.org Received: (qmail 15795 invoked by uid 500); 13 Aug 2013 15:58:44 -0000 Mailing-List: contact user-help@hadoop.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@hadoop.apache.org Delivered-To: mailing list user@hadoop.apache.org Received: (qmail 15787 invoked by uid 99); 13 Aug 2013 15:58:44 -0000 Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 13 Aug 2013 15:58:44 +0000 X-ASF-Spam-Status: No, hits=1.8 required=5.0 tests=FREEMAIL_ENVFROM_END_DIGIT,HTML_MESSAGE,NORMAL_HTTP_TO_IP,RCVD_IN_DNSWL_LOW,SPF_PASS,WEIRD_PORT X-Spam-Check-By: apache.org Received-SPF: pass (nike.apache.org: domain of vkjk89@gmail.com designates 209.85.220.171 as permitted sender) Received: from [209.85.220.171] (HELO mail-vc0-f171.google.com) (209.85.220.171) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 13 Aug 2013 15:58:37 +0000 Received: by mail-vc0-f171.google.com with SMTP id ij15so4070232vcb.16 for ; Tue, 13 Aug 2013 08:58:16 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :content-type; bh=V9AUzMWJw6fGZjL8lppCzJnDNzhF/9nVvRjnDiTV8WU=; b=JqPSOiC3BAsrFlMsAZ4FQKenUC/heRMWyovtNk/3+dTYVvSoT9jk9CFmGERPHynLMV NuQ0lt6ABv3nAbiRz93vgwBKSjnWgv/nkXD90YNxk40T7QWvXirhgfHUMAMxoneQodda sR2eEdsBEV0ZvGUQO9M7Y28sXu+Rp4SNbDS5yeDNbRLmFkvzSqIJis6BG9JZW5zSnrCa X8UhDhGfoPjF9SEgeppBX5pnnxfHv5EOqX/L69QBVYFd0q71y8dJiWDxdf7+tBm+YonY xkm/1Tv2eti14ASdkunNbw9j6SHYOB1tnmRZg5UcshjN0MbY6xch94a1jTeHQ4Y1WVR8 ok/g== MIME-Version: 1.0 X-Received: by 10.52.164.16 with SMTP id ym16mr4106000vdb.32.1376409496492; Tue, 13 Aug 2013 08:58:16 -0700 (PDT) Received: by 10.58.242.6 with HTTP; Tue, 13 Aug 2013 08:58:16 -0700 (PDT) In-Reply-To: References: Date: Tue, 13 Aug 2013 21:28:16 +0530 Message-ID: Subject: Re: Exceptions in Name node and Data node logs From: Vimal Jain To: user@hadoop.apache.org Content-Type: multipart/alternative; boundary=001a11c22e581aa09004e3d64f25 X-Virus-Checked: Checked by ClamAV on apache.org --001a11c22e581aa09004e3d64f25 Content-Type: text/plain; charset=ISO-8859-1 Hi, As Jitendra pointed out , this issue was fixed in .20 version. I am using Hadoop 1.1.2 so why its occurring again ? Please help here. On Tue, Aug 13, 2013 at 2:56 PM, Vimal Jain wrote: > Hi Jitendra, > Thanks for your reply. > Currently my hadoop/hbase is down in production as it had filled up the > disk space with above exceptions in log files and had to be brought down. > Also i am using hadoop/hbase in pseudo distributed mode , so there is > only one node which hosts all 6 processes ( 3 from hadoop and 3 from hbase). > > > > On Tue, Aug 13, 2013 at 2:50 PM, Jitendra Yadav < > jeetuyadav200890@gmail.com> wrote: > >> Hi, >> >> One of your DN is marked as dead because NN is not able to get heartbeat >> message from DN but NN still getting block information from dead node. This >> error is similar to a bug *HDFS-1250* reported 2 years back and fixed in >> 0.20 release. >> >> Can you please check the status of DN's in cluster. >> >> #bin/hadoop dfsadmin -report >> >> Thanks >> >> On Tue, Aug 13, 2013 at 1:53 PM, Vimal Jain wrote: >> >>> Hi, >>> I have configured Hadoop and Hbase in pseudo distributed mode. >>> So far things were working fine , but suddenly i started receiving some >>> exceptions in my namenode and datanode log files. >>> It keeps repeating and thus fills up my disk space. >>> Please help here. >>> >>> *Exception in data node :-* >>> >>> 2013-07-31 19:39:51,094 WARN >>> org.apache.hadoop.hdfs.server.datanode.DataNode: >>> org.apache.hadoop.ipc.RemoteException: java.io.IOException: Got blockRec >>> eived message from unregistered or dead node >>> blk_-4787262105551508952_28369 >>> at >>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.blockReceived(FSNamesystem.java:4188) >>> at >>> org.apache.hadoop.hdfs.server.namenode.NameNode.blockReceived(NameNode.java:1069) >>> at sun.reflect.GeneratedMethodAccessor56.invoke(Unknown Source) >>> at >>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) >>> at java.lang.reflect.Method.invoke(Method.java:597) >>> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:578) >>> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1393) >>> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1389) >>> at java.security.AccessController.doPrivileged(Native Method) >>> at javax.security.auth.Subject.doAs(Subject.java:396) >>> at >>> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1149) >>> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1387) >>> >>> at org.apache.hadoop.ipc.Client.call(Client.java:1107) >>> at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:229) >>> at sun.proxy.$Proxy5.blockReceived(Unknown Source) >>> at >>> org.apache.hadoop.hdfs.server.datanode.DataNode.offerService(DataNode.java:1006) >>> at >>> org.apache.hadoop.hdfs.server.datanode.DataNode.run(DataNode.java:1527) >>> at java.lang.Thread.run(Thread.java:662) >>> >>> *Exception in name node :- * >>> >>> 2013-07-31 19:39:50,671 WARN org.apache.hadoop.hdfs.StateChange: BLOCK* >>> NameSystem.blockReceived: blk_-4787262105551508952_28369 is received from >>> dead >>> or unregistered node 192.168.20.30:50010 >>> 2013-07-31 19:39:50,671 ERROR >>> org.apache.hadoop.security.UserGroupInformation: PriviledgedActionException >>> as:hadoop cause:java.io.IOException: Got blo >>> ckReceived message from unregistered or dead node >>> blk_-4787262105551508952_28369 >>> 2013-07-31 19:39:50,671 INFO org.apache.hadoop.ipc.Server: IPC Server >>> handler 5 on 9000, call blockReceived(DatanodeRegistration( >>> 192.168.20.30:50010, >>> storageID=DS-1816106352-192.168.20.30-50010-1369314076237, >>> infoPort=50075, ipcPort=50020), >>> [Lorg.apache.hadoop.hdfs.protocol.Block;@64f2d559, [Ljava.l >>> ang.String;@294f9d6) from 192.168.20.30:59764: error: >>> java.io.IOException: Got blockReceived message from unregistered or dead >>> node blk_-4787262105551 >>> 508952_28369 >>> java.io.IOException: Got blockReceived message from unregistered or dead >>> node blk_-4787262105551508952_28369 >>> at >>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.blockReceived(FSNamesystem.java:4188) >>> at >>> org.apache.hadoop.hdfs.server.namenode.NameNode.blockReceived(NameNode.java:1069) >>> at sun.reflect.GeneratedMethodAccessor56.invoke(Unknown Source) >>> at >>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) >>> at java.lang.reflect.Method.invoke(Method.java:597) >>> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:578) >>> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1393) >>> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1389) >>> at java.security.AccessController.doPrivileged(Native Method) >>> at javax.security.auth.Subject.doAs(Subject.java:396) >>> at >>> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1149) >>> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1387) >>> >>> >>> -- >>> Thanks and Regards, >>> Vimal Jain >>> >> >> > > > -- > Thanks and Regards, > Vimal Jain > -- Thanks and Regards, Vimal Jain --001a11c22e581aa09004e3d64f25 Content-Type: text/html; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable
Hi,
As Jitendra pointed out , this issue was fixed in = .20 version.
I am using Hadoop 1.1.2 so why its occurring again ?=
Please help here.


=
On Tue, Aug 13, 2013 at 2:56 PM, Vimal Jain <vkjk89@gmail.com> wrote:
Hi Jitendra,
Thanks for your reply= .
Currently my hadoop/hbase is down in production as it had filled= up the disk space with above exceptions in log files=A0 and had to be brou= ght down.
Also i am using hadoop/hbase=A0 in pseudo distributed mode , so there= is only one node which hosts all 6 processes ( 3 from hadoop and 3 from hb= ase).



On Tue, Aug 13, 2013 at 2:50 PM, Jitendra Yadav <jeetuyadav200890= @gmail.com> wrote:
Hi,
=A0
One of=A0your DN is marked as dead because NN is not able to get heart= beat message from DN but NN still getting block information from dead node.= =A0This error is similar to a bug HDFS-1250=A0reported 2 y= ears back and fixed in 0.20 release.
=A0
Can you=A0please=A0check the status of DN's in=A0cluster.
=A0
#bin/hadoop dfsadmin -report

Thanks

On Tue, Aug 13, 2013 at 1:53 PM, Vimal Jain <vkjk89= @gmail.com> wrote:
Hi,
I have configured Hadoop and Hbase in pseudo distributed mode.<= br>
So far things were working fine , but suddenly i started receiving= some exceptions in my namenode and datanode log files.
It keeps r= epeating and thus fills up my disk space.
Please help here.

Exception in data node :-
2013-07-31 19:39:51,094 WARN org.apache.hadoop.hdfs.server.datanode.DataNo= de: org.apache.hadoop.ipc.RemoteException: java.io.IOException: Got blockRe= c
eived message from unregistered or dead node blk_-4787262105551508952_28369=
=A0=A0=A0 at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.blockR= eceived(FSNamesystem.java:4188)
=A0=A0=A0 at org.apache.hadoop.hdfs.serv= er.namenode.NameNode.blockReceived(NameNode.java:1069)
=A0=A0=A0 at sun.reflect.GeneratedMethodAccessor56.invoke(Unknown Source)=A0=A0=A0 at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMe= thodAccessorImpl.java:25)
=A0=A0=A0 at java.lang.reflect.Method.invoke(M= ethod.java:597)
=A0=A0=A0 at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:578)
=A0=A0= =A0 at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1393)
=A0= =A0=A0 at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1389)
= =A0=A0=A0 at java.security.AccessController.doPrivileged(Native Method)
=A0=A0=A0 at javax.security.auth.Subject.doAs(Subject.java:396)
=A0=A0= =A0 at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInform= ation.java:1149)
=A0=A0=A0 at org.apache.hadoop.ipc.Server$Handler.run(S= erver.java:1387)

=A0=A0=A0 at org.apache.hadoop.ipc.Client.call(Client.java:1107)
=A0= =A0=A0 at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:229)
=A0=A0= =A0 at sun.proxy.$Proxy5.blockReceived(Unknown Source)
=A0=A0=A0 at org.= apache.hadoop.hdfs.server.datanode.DataNode.offerService(DataNode.java:1006= )
=A0=A0=A0 at org.apache.hadoop.hdfs.server.datanode.DataNode.run(DataNode.j= ava:1527)
=A0=A0=A0 at java.lang.Thread.run(Thread.java:662)

Exception in name node :-

2013-07-31 19:39:50,671 WARN org= .apache.hadoop.hdfs.StateChange: BLOCK* NameSystem.blockReceived: blk_-4787= 262105551508952_28369 is received from dead
=A0or unregistered node 192.168.20.30:50010
2013-07-31 19:39:50,671 ERROR org.apache.= hadoop.security.UserGroupInformation: PriviledgedActionException as:hadoop = cause:java.io.IOException: Got blo
ckReceived message from unregistered or dead node blk_-4787262105551508952_= 28369
2013-07-31 19:39:50,671 INFO org.apache.hadoop.ipc.Server: IPC Ser= ver handler 5 on 9000, call blockReceived(DatanodeRegistration(192.168.20.30:50010,
storageID=3DDS-1816106352-192.168.20.30-50010-1369314076237, infoPort=3D500= 75, ipcPort=3D50020), [Lorg.apache.hadoop.hdfs.protocol.Block;@64f2d559, [L= java.l
ang.String;@294f9d6) from 192.168.20.30:59764: error: java.io.IOException: Got= blockReceived message from unregistered or dead node blk_-4787262105551 508952_28369
java.io.IOException: Got blockReceived message from unregis= tered or dead node blk_-4787262105551508952_28369
=A0=A0=A0 at org.apach= e.hadoop.hdfs.server.namenode.FSNamesystem.blockReceived(FSNamesystem.java:= 4188)
=A0=A0=A0 at org.apache.hadoop.hdfs.server.namenode.NameNode.blockReceived(= NameNode.java:1069)
=A0=A0=A0 at sun.reflect.GeneratedMethodAccessor56.i= nvoke(Unknown Source)
=A0=A0=A0 at sun.reflect.DelegatingMethodAccessorI= mpl.invoke(DelegatingMethodAccessorImpl.java:25)
=A0=A0=A0 at java.lang.reflect.Method.invoke(Method.java:597)
=A0=A0=A0 = at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:578)
=A0=A0=A0 at org.= apache.hadoop.ipc.Server$Handler$1.run(Server.java:1393)
=A0=A0=A0 at or= g.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1389)
=A0=A0=A0 at java.security.AccessController.doPrivileged(Native Method)
= =A0=A0=A0 at javax.security.auth.Subject.doAs(Subject.java:396)
=A0=A0= =A0 at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInform= ation.java:1149)
=A0=A0=A0 at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1387)


--
Thanks and Regards,<= /font>
Vimal Jain




--
Thanks and Regards,
Vimal Jain<= /div>



--
=
Thanks and Regards,<= /font>
Vimal Ja= in
--001a11c22e581aa09004e3d64f25--