Return-Path: X-Original-To: apmail-hadoop-user-archive@minotaur.apache.org Delivered-To: apmail-hadoop-user-archive@minotaur.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 74748104DD for ; Fri, 16 Aug 2013 10:46:01 +0000 (UTC) Received: (qmail 48019 invoked by uid 500); 16 Aug 2013 10:45:56 -0000 Delivered-To: apmail-hadoop-user-archive@hadoop.apache.org Received: (qmail 47686 invoked by uid 500); 16 Aug 2013 10:45:55 -0000 Mailing-List: contact user-help@hadoop.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@hadoop.apache.org Delivered-To: mailing list user@hadoop.apache.org Received: (qmail 47677 invoked by uid 99); 16 Aug 2013 10:45:55 -0000 Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 16 Aug 2013 10:45:55 +0000 X-ASF-Spam-Status: No, hits=-1.8 required=5.0 tests=HTML_MESSAGE,NORMAL_HTTP_TO_IP,RCVD_IN_DNSWL_HI,SPF_SOFTFAIL,WEIRD_PORT X-Spam-Check-By: apache.org Received-SPF: softfail (athena.apache.org: transitioning domain of hadoophive@gmail.com does not designate 17.151.62.50 as permitted sender) Received: from [17.151.62.50] (HELO mail-out.apple.com) (17.151.62.50) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 16 Aug 2013 10:45:47 +0000 MIME-version: 1.0 Content-type: multipart/alternative; boundary="Boundary_(ID_ah2lvsrWJ8UX3BQffaQzDw)" Received: from relay8.apple.com ([17.128.113.102]) by mail-out.apple.com (Oracle Communications Messaging Server 7u4-23.01 (7.0.4.23.0) 64bit (built Aug 10 2011)) with ESMTP id <0MRM00IBYDVKEMN1@mail-out.apple.com> for user@hadoop.apache.org; Fri, 16 Aug 2013 03:45:27 -0700 (PDT) X-AuditID: 11807166-b7fbc6d000002e73-a1-520e02c6589e Received: from orrisroot.apple.com (orrisroot.apple.com [17.128.115.106]) (using TLS with cipher RC4-MD5 (128/128 bits)) (Client did not present a certificate) by relay8.apple.com (Apple SCV relay) with SMTP id FD.8A.11891.6C20E025; Fri, 16 Aug 2013 03:45:26 -0700 (PDT) Received: from [172.21.225.97] (unknown [17.151.191.15]) by orrisroot.apple.com (Oracle Communications Messaging Server 7u4-24.01 (7.0.4.24.0) 64bit (built Nov 17 2011)) with ESMTPSA id <0MRM0095CDVMFP50@orrisroot.apple.com> for user@hadoop.apache.org; Fri, 16 Aug 2013 03:45:26 -0700 (PDT) From: t578384 Message-id: Subject: Re: Exceptions in Name node and Data node logs Date: Fri, 16 Aug 2013 03:45:21 -0700 References: To: user@hadoop.apache.org In-reply-to: X-Mailer: Apple Mail (2.1508) X-Brightmail-Tracker: H4sIAAAAAAAAA+NgFtrILMWRmVeSWpSXmKPExsUi2FCcpXuMiS/IYE+buUXPlGksDoweE7q2 MAYwRnHZpKTmZJalFunbJXBlnDv6jLFgS23FjZ9/GRsYV+d2MXJySAiYSFxecJkZwhaTuHBv PVsXIxeHkMAUJom3z+9COauYJH7seMYEUsUmoCJxrWcemM0skCTRvW4tG4jNK2Aj0d9wihHC 1pPYPOMFO4gtLGAusfnfdaAaDg4WAVWJy3f0IWa+Y5LY/f8AC0iNiICURPebyWAzOQWCJbb1 v2YGqZcQkJXY+TtpAiPfLCTbZiHZABHXlli28DUzhK0n8bLpHRZxXYmL6yYxLmBkW8UoUJSa k1hpoZdYUJCTqpecn7uJERyQhWk7GJuWWx1iFOBgVOLhZZjIGyTEmlhWXJl7iFGCg1lJhHfK Q6AQb0piZVVqUX58UWlOavEhRmkOFiVx3gWHgFIC6YklqdmpqQWpRTBZJg5OqQZGDcOsd5cv LwvZ88eif5tP9oycU9s2p7dfqbM8/8Up4kaivYTNtj8bVuXPZtmW58DwfK8s85oCoyv/hY5P K90jUisi/3Gmqkk0x+/cP5Ol/Jlq17YFRXyyyY7P2ziBK2GejrD1q0+WnVULc7j3P532/+sx NQaHWYIHJuUsWrA/VynNwk4hviBHiaU4I9FQi7moOBEAq9UXCEQCAAA= X-Virus-Checked: Checked by ClamAV on apache.org --Boundary_(ID_ah2lvsrWJ8UX3BQffaQzDw) Content-type: text/plain; CHARSET=US-ASCII Content-transfer-encoding: 7BIT Hi Vimal , Could you elaborate this , what do you mean running 6 process in on single node, How many nodes do you have in total and all are used for hadoop and hbase as well. Possibilities: - Might be there is dn no listed in your slave file and configure as DN in you cluster. chances are your Nodes and not configure properly thanks Vikas Srivastava On Aug 15, 2013, at 10:37 PM, Vimal Jain wrote: > Can someone please help here ? > > > On Tue, Aug 13, 2013 at 9:28 PM, Vimal Jain wrote: > Hi, > As Jitendra pointed out , this issue was fixed in .20 version. > I am using Hadoop 1.1.2 so why its occurring again ? > Please help here. > > > On Tue, Aug 13, 2013 at 2:56 PM, Vimal Jain wrote: > Hi Jitendra, > Thanks for your reply. > Currently my hadoop/hbase is down in production as it had filled up the disk space with above exceptions in log files and had to be brought down. > Also i am using hadoop/hbase in pseudo distributed mode , so there is only one node which hosts all 6 processes ( 3 from hadoop and 3 from hbase). > > > > On Tue, Aug 13, 2013 at 2:50 PM, Jitendra Yadav wrote: > Hi, > > One of your DN is marked as dead because NN is not able to get heartbeat message from DN but NN still getting block information from dead node. This error is similar to a bug HDFS-1250 reported 2 years back and fixed in 0.20 release. > > Can you please check the status of DN's in cluster. > > #bin/hadoop dfsadmin -report > > Thanks > > On Tue, Aug 13, 2013 at 1:53 PM, Vimal Jain wrote: > Hi, > I have configured Hadoop and Hbase in pseudo distributed mode. > So far things were working fine , but suddenly i started receiving some exceptions in my namenode and datanode log files. > It keeps repeating and thus fills up my disk space. > Please help here. > > Exception in data node :- > > 2013-07-31 19:39:51,094 WARN org.apache.hadoop.hdfs.server.datanode.DataNode: org.apache.hadoop.ipc.RemoteException: java.io.IOException: Got blockRec > eived message from unregistered or dead node blk_-4787262105551508952_28369 > at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.blockReceived(FSNamesystem.java:4188) > at org.apache.hadoop.hdfs.server.namenode.NameNode.blockReceived(NameNode.java:1069) > at sun.reflect.GeneratedMethodAccessor56.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) > at java.lang.reflect.Method.invoke(Method.java:597) > at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:578) > at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1393) > at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1389) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:396) > at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1149) > at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1387) > > at org.apache.hadoop.ipc.Client.call(Client.java:1107) > at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:229) > at sun.proxy.$Proxy5.blockReceived(Unknown Source) > at org.apache.hadoop.hdfs.server.datanode.DataNode.offerService(DataNode.java:1006) > at org.apache.hadoop.hdfs.server.datanode.DataNode.run(DataNode.java:1527) > at java.lang.Thread.run(Thread.java:662) > > Exception in name node :- > > 2013-07-31 19:39:50,671 WARN org.apache.hadoop.hdfs.StateChange: BLOCK* NameSystem.blockReceived: blk_-4787262105551508952_28369 is received from dead > or unregistered node 192.168.20.30:50010 > 2013-07-31 19:39:50,671 ERROR org.apache.hadoop.security.UserGroupInformation: PriviledgedActionException as:hadoop cause:java.io.IOException: Got blo > ckReceived message from unregistered or dead node blk_-4787262105551508952_28369 > 2013-07-31 19:39:50,671 INFO org.apache.hadoop.ipc.Server: IPC Server handler 5 on 9000, call blockReceived(DatanodeRegistration(192.168.20.30:50010, > storageID=DS-1816106352-192.168.20.30-50010-1369314076237, infoPort=50075, ipcPort=50020), [Lorg.apache.hadoop.hdfs.protocol.Block;@64f2d559, [Ljava.l > ang.String;@294f9d6) from 192.168.20.30:59764: error: java.io.IOException: Got blockReceived message from unregistered or dead node blk_-4787262105551 > 508952_28369 > java.io.IOException: Got blockReceived message from unregistered or dead node blk_-4787262105551508952_28369 > at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.blockReceived(FSNamesystem.java:4188) > at org.apache.hadoop.hdfs.server.namenode.NameNode.blockReceived(NameNode.java:1069) > at sun.reflect.GeneratedMethodAccessor56.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) > at java.lang.reflect.Method.invoke(Method.java:597) > at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:578) > at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1393) > at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1389) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:396) > at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1149) > at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1387) > > > -- > Thanks and Regards, > Vimal Jain > > > > > -- > Thanks and Regards, > Vimal Jain > > > > -- > Thanks and Regards, > Vimal Jain > > > > -- > Thanks and Regards, > Vimal Jain --Boundary_(ID_ah2lvsrWJ8UX3BQffaQzDw) Content-type: text/html; CHARSET=US-ASCII Content-transfer-encoding: quoted-printable Hi = Vimal ,

Could you elaborate this , what do you mean = running 6 process in on single node,

How many = nodes do you have in total and all are used for hadoop and hbase as = well.

Possibilities: - 
Might be = there is dn no listed in your slave file and configure as DN in you = cluster.
chances are your Nodes and not configure = properly

thanks
Vikas = Srivastava


On Aug 15, 2013, at = 10:37 PM, Vimal Jain <vkjk89@gmail.com> = wrote:

Can someone please help here = ?


On Tue, Aug 13, 2013 at 9:28 PM, Vimal Jain <vkjk89@gmail.com> wrote:
Hi,
As Jitendra pointed out , this issue was fixed in = .20 version.
I am using Hadoop 1.1.2 so why its occurring = again ?
Please help here.


On Tue, Aug 13, 2013 at 2:56 PM, Vimal Jain <vkjk89@gmail.com> wrote:
Hi Jitendra,
Thanks for your = reply.
Currently my hadoop/hbase is down in production as it = had filled up the disk space with above exceptions in log files  = and had to be brought down.
Also i am using hadoop/hbase  in pseudo distributed mode , so = there is only one node which hosts all 6 processes ( 3 from hadoop and 3 = from hbase).



On Tue, Aug 13, 2013 at 2:50 PM, Jitendra Yadav <jeetuyadav200890@gmail.com> = wrote:
Hi,
 
One of your DN is marked as dead because NN is not able to get = heartbeat message from DN but NN still getting block information from = dead node. This error is similar to a bug = HDFS-1250 reported 2 years back and fixed in 0.20 = release.
 
Can you please check the status of DN's = in cluster.
 
#bin/hadoop dfsadmin -report

Thanks

On Tue, Aug 13, 2013 at 1:53 PM, Vimal Jain = <vkjk89@gmail.com> = wrote:
Hi,
I have configured Hadoop and Hbase in pseudo distributed = mode.
So far things were working fine , but suddenly i started = receiving some exceptions in my namenode and datanode log = files.
It keeps repeating and thus fills up my disk space.
Please help here.

Exception in data node = :-

2013-07-31 19:39:51,094 WARN = org.apache.hadoop.hdfs.server.datanode.DataNode: = org.apache.hadoop.ipc.RemoteException: java.io.IOException: Got = blockRec
eived message from unregistered or dead node = blk_-4787262105551508952_28369
    at = org.apache.hadoop.hdfs.server.namenode.FSNamesystem.blockReceived(FSNamesy= stem.java:4188)
    at = org.apache.hadoop.hdfs.server.namenode.NameNode.blockReceived(NameNode.jav= a:1069)
    at = sun.reflect.GeneratedMethodAccessor56.invoke(Unknown = Source)
    at = sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorIm= pl.java:25)
    at = java.lang.reflect.Method.invoke(Method.java:597)
    at = org.apache.hadoop.ipc.RPC$Server.call(RPC.java:578)
    = at = org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1393)
 &nbs= p;  at = org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1389)
 &nbs= p;  at java.security.AccessController.doPrivileged(Native = Method)
    at = javax.security.auth.Subject.doAs(Subject.java:396)
    = at = org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.= java:1149)
    at = org.apache.hadoop.ipc.Server$Handler.run(Server.java:1387)

    at = org.apache.hadoop.ipc.Client.call(Client.java:1107)
    = at = org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:229)
  &nbs= p; at sun.proxy.$Proxy5.blockReceived(Unknown = Source)
    at = org.apache.hadoop.hdfs.server.datanode.DataNode.offerService(DataNode.java= :1006)
    at = org.apache.hadoop.hdfs.server.datanode.DataNode.run(DataNode.java:1527)    at = java.lang.Thread.run(Thread.java:662)

Exception in name = node :-

2013-07-31 19:39:50,671 WARN = org.apache.hadoop.hdfs.StateChange: BLOCK* NameSystem.blockReceived: = blk_-4787262105551508952_28369 is received from dead
 or unregistered node 192.168.20.30:50010
2013-07-31 19:39:50,671 = ERROR org.apache.hadoop.security.UserGroupInformation: = PriviledgedActionException as:hadoop cause:java.io.IOException: Got = blo
ckReceived message from unregistered or dead node = blk_-4787262105551508952_28369
2013-07-31 19:39:50,671 INFO = org.apache.hadoop.ipc.Server: IPC Server handler 5 on 9000, call = blockReceived(DatanodeRegistration(192.168.20.30:50010,
storageID=3DDS-1816106352-192.168.20.30-50010-1369314076237, = infoPort=3D50075, ipcPort=3D50020), = [Lorg.apache.hadoop.hdfs.protocol.Block;@64f2d559, = [Ljava.l
ang.String;@294f9d6) from 192.168.20.30:59764: error: java.io.IOException: = Got blockReceived message from unregistered or dead node = blk_-4787262105551
508952_28369
java.io.IOException: Got blockReceived message from = unregistered or dead node = blk_-4787262105551508952_28369
    at = org.apache.hadoop.hdfs.server.namenode.FSNamesystem.blockReceived(FSNamesy= stem.java:4188)
    at = org.apache.hadoop.hdfs.server.namenode.NameNode.blockReceived(NameNode.jav= a:1069)
    at = sun.reflect.GeneratedMethodAccessor56.invoke(Unknown = Source)
    at = sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorIm= pl.java:25)
    at = java.lang.reflect.Method.invoke(Method.java:597)
    = at = org.apache.hadoop.ipc.RPC$Server.call(RPC.java:578)
    = at = org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1393)
 &nbs= p;  at = org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1389)
    at java.security.AccessController.doPrivileged(Native = Method)
    at = javax.security.auth.Subject.doAs(Subject.java:396)
    = at = org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.= java:1149)
    at = org.apache.hadoop.ipc.Server$Handler.run(Server.java:1387)


--
Thanks and = Regards,
Vimal = Jain
<= /blockquote>




--
Thanks and = Regards,
Vimal Jain



-- =
Thanks and = Regards,
Vimal Jain



--
Thanks and = Regards,
Vimal Jain

= --Boundary_(ID_ah2lvsrWJ8UX3BQffaQzDw)--