Return-Path: X-Original-To: apmail-hadoop-common-user-archive@www.apache.org Delivered-To: apmail-hadoop-common-user-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 336F8F3EB for ; Tue, 13 Aug 2013 09:20:54 +0000 (UTC) Received: (qmail 8937 invoked by uid 500); 13 Aug 2013 09:20:45 -0000 Delivered-To: apmail-hadoop-common-user-archive@hadoop.apache.org Received: (qmail 8571 invoked by uid 500); 13 Aug 2013 09:20:42 -0000 Mailing-List: contact user-help@hadoop.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@hadoop.apache.org Delivered-To: mailing list user@hadoop.apache.org Received: (qmail 8554 invoked by uid 99); 13 Aug 2013 09:20:40 -0000 Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 13 Aug 2013 09:20:40 +0000 X-ASF-Spam-Status: No, hits=1.8 required=5.0 tests=FREEMAIL_ENVFROM_END_DIGIT,HTML_MESSAGE,NORMAL_HTTP_TO_IP,RCVD_IN_DNSWL_LOW,SPF_PASS,WEIRD_PORT X-Spam-Check-By: apache.org Received-SPF: pass (nike.apache.org: domain of jeetuyadav200890@gmail.com designates 209.85.217.171 as permitted sender) Received: from [209.85.217.171] (HELO mail-lb0-f171.google.com) (209.85.217.171) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 13 Aug 2013 09:20:33 +0000 Received: by mail-lb0-f171.google.com with SMTP id t13so5620056lbd.16 for ; Tue, 13 Aug 2013 02:20:12 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :content-type; bh=1uJBLg2r1wsjqxHYoKPI7/dp9IwQrHjhqVra4h9aP8s=; b=YuC2bgBdwLyrrjxcDXvfoQ74+5SbxKT24XKRulPcdAZHNCvrok3xN8nCXkIo59pAP0 OJ9GTK4nYqjCtggN0WTlFQHdgp2E9IBCMPa14pJXXRQvhVY6w5ONLgP4nUf9nDc6+92l 4KdQlxEanrI7SWKvhX/S+o0+8RsGKBuFJjXY3RRkHzoxQyha0/NunimgIxEEO/3UG244 W9S3KwU7k4VobFHTtPo7UWJaY8PGUXQ44rCxTj5qNY9s1BjPPiAxmun1dfvYDn8MKPoc 2bk5j0ILkrwj+GGj0LeHKpmXUEnsKYvtQRUZirZtnkARHP3ijTWgxbUFRP0zqZJ7yMui JF1w== MIME-Version: 1.0 X-Received: by 10.112.74.20 with SMTP id p20mr868626lbv.36.1376385612688; Tue, 13 Aug 2013 02:20:12 -0700 (PDT) Received: by 10.112.180.70 with HTTP; Tue, 13 Aug 2013 02:20:12 -0700 (PDT) In-Reply-To: References: Date: Tue, 13 Aug 2013 14:50:12 +0530 Message-ID: Subject: Re: Exceptions in Name node and Data node logs From: Jitendra Yadav To: user@hadoop.apache.org Content-Type: multipart/alternative; boundary=14dae9473c2b84b3ed04e3d0bfe8 X-Virus-Checked: Checked by ClamAV on apache.org --14dae9473c2b84b3ed04e3d0bfe8 Content-Type: text/plain; charset=ISO-8859-1 Hi, One of your DN is marked as dead because NN is not able to get heartbeat message from DN but NN still getting block information from dead node. This error is similar to a bug *HDFS-1250* reported 2 years back and fixed in 0.20 release. Can you please check the status of DN's in cluster. #bin/hadoop dfsadmin -report Thanks On Tue, Aug 13, 2013 at 1:53 PM, Vimal Jain wrote: > Hi, > I have configured Hadoop and Hbase in pseudo distributed mode. > So far things were working fine , but suddenly i started receiving some > exceptions in my namenode and datanode log files. > It keeps repeating and thus fills up my disk space. > Please help here. > > *Exception in data node :-* > > 2013-07-31 19:39:51,094 WARN > org.apache.hadoop.hdfs.server.datanode.DataNode: > org.apache.hadoop.ipc.RemoteException: java.io.IOException: Got blockRec > eived message from unregistered or dead node blk_-4787262105551508952_28369 > at > org.apache.hadoop.hdfs.server.namenode.FSNamesystem.blockReceived(FSNamesystem.java:4188) > at > org.apache.hadoop.hdfs.server.namenode.NameNode.blockReceived(NameNode.java:1069) > at sun.reflect.GeneratedMethodAccessor56.invoke(Unknown Source) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) > at java.lang.reflect.Method.invoke(Method.java:597) > at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:578) > at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1393) > at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1389) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:396) > at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1149) > at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1387) > > at org.apache.hadoop.ipc.Client.call(Client.java:1107) > at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:229) > at sun.proxy.$Proxy5.blockReceived(Unknown Source) > at > org.apache.hadoop.hdfs.server.datanode.DataNode.offerService(DataNode.java:1006) > at > org.apache.hadoop.hdfs.server.datanode.DataNode.run(DataNode.java:1527) > at java.lang.Thread.run(Thread.java:662) > > *Exception in name node :- * > > 2013-07-31 19:39:50,671 WARN org.apache.hadoop.hdfs.StateChange: BLOCK* > NameSystem.blockReceived: blk_-4787262105551508952_28369 is received from > dead > or unregistered node 192.168.20.30:50010 > 2013-07-31 19:39:50,671 ERROR > org.apache.hadoop.security.UserGroupInformation: PriviledgedActionException > as:hadoop cause:java.io.IOException: Got blo > ckReceived message from unregistered or dead node > blk_-4787262105551508952_28369 > 2013-07-31 19:39:50,671 INFO org.apache.hadoop.ipc.Server: IPC Server > handler 5 on 9000, call blockReceived(DatanodeRegistration( > 192.168.20.30:50010, > storageID=DS-1816106352-192.168.20.30-50010-1369314076237, infoPort=50075, > ipcPort=50020), [Lorg.apache.hadoop.hdfs.protocol.Block;@64f2d559, > [Ljava.l > ang.String;@294f9d6) from 192.168.20.30:59764: error: > java.io.IOException: Got blockReceived message from unregistered or dead > node blk_-4787262105551 > 508952_28369 > java.io.IOException: Got blockReceived message from unregistered or dead > node blk_-4787262105551508952_28369 > at > org.apache.hadoop.hdfs.server.namenode.FSNamesystem.blockReceived(FSNamesystem.java:4188) > at > org.apache.hadoop.hdfs.server.namenode.NameNode.blockReceived(NameNode.java:1069) > at sun.reflect.GeneratedMethodAccessor56.invoke(Unknown Source) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) > at java.lang.reflect.Method.invoke(Method.java:597) > at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:578) > at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1393) > at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1389) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:396) > at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1149) > at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1387) > > > -- > Thanks and Regards, > Vimal Jain > --14dae9473c2b84b3ed04e3d0bfe8 Content-Type: text/html; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable
Hi,
=A0
One of=A0your DN is marked as dead because NN is not able to get heart= beat message from DN but NN still getting block information from dead node.= =A0This error is similar to a bug HDFS-1250=A0reported 2 y= ears back and fixed in 0.20 release.
=A0
Can you=A0please=A0check the status of DN's in=A0cluster.
=A0
#bin/hadoop dfsadmin -report

Thanks
On Tue, Aug 13, 2013 at 1:53 PM, Vimal Jain <vkjk89@gmai= l.com> wrote:
Hi,
I have configured Hadoop and Hbase in pseudo distributed mode.<= br>
So far things were working fine , but suddenly i started receiving= some exceptions in my namenode and datanode log files.
It keeps r= epeating and thus fills up my disk space.
Please help here.

Exception in data node :-
2013-07-31 19:39:51,094 WARN org.apache.hadoop.hdfs.server.datanode.DataNo= de: org.apache.hadoop.ipc.RemoteException: java.io.IOException: Got blockRe= c
eived message from unregistered or dead node blk_-4787262105551508952_28369=
=A0=A0=A0 at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.blockR= eceived(FSNamesystem.java:4188)
=A0=A0=A0 at org.apache.hadoop.hdfs.serv= er.namenode.NameNode.blockReceived(NameNode.java:1069)
=A0=A0=A0 at sun.reflect.GeneratedMethodAccessor56.invoke(Unknown Source)=A0=A0=A0 at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMe= thodAccessorImpl.java:25)
=A0=A0=A0 at java.lang.reflect.Method.invoke(M= ethod.java:597)
=A0=A0=A0 at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:578)
=A0=A0= =A0 at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1393)
=A0= =A0=A0 at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1389)
= =A0=A0=A0 at java.security.AccessController.doPrivileged(Native Method)
=A0=A0=A0 at javax.security.auth.Subject.doAs(Subject.java:396)
=A0=A0= =A0 at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInform= ation.java:1149)
=A0=A0=A0 at org.apache.hadoop.ipc.Server$Handler.run(S= erver.java:1387)

=A0=A0=A0 at org.apache.hadoop.ipc.Client.call(Client.java:1107)
=A0= =A0=A0 at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:229)
=A0=A0= =A0 at sun.proxy.$Proxy5.blockReceived(Unknown Source)
=A0=A0=A0 at org.= apache.hadoop.hdfs.server.datanode.DataNode.offerService(DataNode.java:1006= )
=A0=A0=A0 at org.apache.hadoop.hdfs.server.datanode.DataNode.run(DataNode.j= ava:1527)
=A0=A0=A0 at java.lang.Thread.run(Thread.java:662)

Exception in name node :-

2013-07-31 19:39:50,671 WARN org= .apache.hadoop.hdfs.StateChange: BLOCK* NameSystem.blockReceived: blk_-4787= 262105551508952_28369 is received from dead
=A0or unregistered node 192.168.20.30:50010
2013-07-31 19:39:50,671 ERROR org.apache.= hadoop.security.UserGroupInformation: PriviledgedActionException as:hadoop = cause:java.io.IOException: Got blo
ckReceived message from unregistered or dead node blk_-4787262105551508952_= 28369
2013-07-31 19:39:50,671 INFO org.apache.hadoop.ipc.Server: IPC Ser= ver handler 5 on 9000, call blockReceived(DatanodeRegistration(192.168.20.30:50010,
storageID=3DDS-1816106352-192.168.20.30-50010-1369314076237, infoPort=3D500= 75, ipcPort=3D50020), [Lorg.apache.hadoop.hdfs.protocol.Block;@64f2d559, [L= java.l
ang.String;@294f9d6) from 192.168.20.30:59764: error: java.io.IOException: Got= blockReceived message from unregistered or dead node blk_-4787262105551 508952_28369
java.io.IOException: Got blockReceived message from unregis= tered or dead node blk_-4787262105551508952_28369
=A0=A0=A0 at org.apach= e.hadoop.hdfs.server.namenode.FSNamesystem.blockReceived(FSNamesystem.java:= 4188)
=A0=A0=A0 at org.apache.hadoop.hdfs.server.namenode.NameNode.blockReceived(= NameNode.java:1069)
=A0=A0=A0 at sun.reflect.GeneratedMethodAccessor56.i= nvoke(Unknown Source)
=A0=A0=A0 at sun.reflect.DelegatingMethodAccessorI= mpl.invoke(DelegatingMethodAccessorImpl.java:25)
=A0=A0=A0 at java.lang.reflect.Method.invoke(Method.java:597)
=A0=A0=A0 = at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:578)
=A0=A0=A0 at org.= apache.hadoop.ipc.Server$Handler$1.run(Server.java:1393)
=A0=A0=A0 at or= g.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1389)
=A0=A0=A0 at java.security.AccessController.doPrivileged(Native Method)
= =A0=A0=A0 at javax.security.auth.Subject.doAs(Subject.java:396)
=A0=A0= =A0 at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInform= ation.java:1149)
=A0=A0=A0 at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1387)


--
Thanks and Regards,<= /font>
Vimal Jain

--14dae9473c2b84b3ed04e3d0bfe8--