hadoop-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jitendra Yadav <jeetuyadav200...@gmail.com>
Subject Re: Exceptions in Name node and Data node logs
Date Tue, 13 Aug 2013 09:20:12 GMT
Hi,

One of your DN is marked as dead because NN is not able to get heartbeat
message from DN but NN still getting block information from dead node. This
error is similar to a bug *HDFS-1250* reported 2 years back and fixed in
0.20 release.

Can you please check the status of DN's in cluster.

#bin/hadoop dfsadmin -report

Thanks
On Tue, Aug 13, 2013 at 1:53 PM, Vimal Jain <vkjk89@gmail.com> wrote:

>   Hi,
> I have configured Hadoop and Hbase in pseudo distributed mode.
> So far things were working fine , but suddenly i started receiving some
> exceptions in my namenode and datanode log files.
> It keeps repeating and thus fills up my disk space.
> Please help here.
>
> *Exception in data node :-*
>
> 2013-07-31 19:39:51,094 WARN
> org.apache.hadoop.hdfs.server.datanode.DataNode:
> org.apache.hadoop.ipc.RemoteException: java.io.IOException: Got blockRec
> eived message from unregistered or dead node blk_-4787262105551508952_28369
>     at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.blockReceived(FSNamesystem.java:4188)
>     at
> org.apache.hadoop.hdfs.server.namenode.NameNode.blockReceived(NameNode.java:1069)
>     at sun.reflect.GeneratedMethodAccessor56.invoke(Unknown Source)
>     at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>     at java.lang.reflect.Method.invoke(Method.java:597)
>     at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:578)
>     at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1393)
>     at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1389)
>     at java.security.AccessController.doPrivileged(Native Method)
>     at javax.security.auth.Subject.doAs(Subject.java:396)
>     at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1149)
>     at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1387)
>
>     at org.apache.hadoop.ipc.Client.call(Client.java:1107)
>     at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:229)
>     at sun.proxy.$Proxy5.blockReceived(Unknown Source)
>     at
> org.apache.hadoop.hdfs.server.datanode.DataNode.offerService(DataNode.java:1006)
>     at
> org.apache.hadoop.hdfs.server.datanode.DataNode.run(DataNode.java:1527)
>     at java.lang.Thread.run(Thread.java:662)
>
> *Exception in name node :- *
>
> 2013-07-31 19:39:50,671 WARN org.apache.hadoop.hdfs.StateChange: BLOCK*
> NameSystem.blockReceived: blk_-4787262105551508952_28369 is received from
> dead
>  or unregistered node 192.168.20.30:50010
> 2013-07-31 19:39:50,671 ERROR
> org.apache.hadoop.security.UserGroupInformation: PriviledgedActionException
> as:hadoop cause:java.io.IOException: Got blo
> ckReceived message from unregistered or dead node
> blk_-4787262105551508952_28369
> 2013-07-31 19:39:50,671 INFO org.apache.hadoop.ipc.Server: IPC Server
> handler 5 on 9000, call blockReceived(DatanodeRegistration(
> 192.168.20.30:50010,
> storageID=DS-1816106352-192.168.20.30-50010-1369314076237, infoPort=50075,
> ipcPort=50020), [Lorg.apache.hadoop.hdfs.protocol.Block;@64f2d559,
> [Ljava.l
> ang.String;@294f9d6) from 192.168.20.30:59764: error:
> java.io.IOException: Got blockReceived message from unregistered or dead
> node blk_-4787262105551
> 508952_28369
> java.io.IOException: Got blockReceived message from unregistered or dead
> node blk_-4787262105551508952_28369
>     at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.blockReceived(FSNamesystem.java:4188)
>     at
> org.apache.hadoop.hdfs.server.namenode.NameNode.blockReceived(NameNode.java:1069)
>     at sun.reflect.GeneratedMethodAccessor56.invoke(Unknown Source)
>     at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>     at java.lang.reflect.Method.invoke(Method.java:597)
>     at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:578)
>     at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1393)
>     at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1389)
>     at java.security.AccessController.doPrivileged(Native Method)
>     at javax.security.auth.Subject.doAs(Subject.java:396)
>     at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1149)
>     at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1387)
>
>
> --
> Thanks and Regards,
> Vimal Jain
>

Mime
View raw message