hadoop-mapreduce-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From t578384 <hadooph...@gmail.com>
Subject Re: Exceptions in Name node and Data node logs
Date Fri, 16 Aug 2013 10:45:21 GMT
Hi Vimal ,

Could you elaborate this , what do you mean running 6 process in on single node,

How many nodes do you have in total and all are used for hadoop and hbase as well.

Possibilities: - 
Might be there is dn no listed in your slave file and configure as DN in you cluster.
chances are your Nodes and not configure properly

thanks
Vikas Srivastava


On Aug 15, 2013, at 10:37 PM, Vimal Jain <vkjk89@gmail.com> wrote:

> Can someone please help here ?
> 
> 
> On Tue, Aug 13, 2013 at 9:28 PM, Vimal Jain <vkjk89@gmail.com> wrote:
> Hi,
> As Jitendra pointed out , this issue was fixed in .20 version.
> I am using Hadoop 1.1.2 so why its occurring again ?
> Please help here.
> 
> 
> On Tue, Aug 13, 2013 at 2:56 PM, Vimal Jain <vkjk89@gmail.com> wrote:
> Hi Jitendra,
> Thanks for your reply.
> Currently my hadoop/hbase is down in production as it had filled up the disk space with
above exceptions in log files  and had to be brought down.
> Also i am using hadoop/hbase  in pseudo distributed mode , so there is only one node
which hosts all 6 processes ( 3 from hadoop and 3 from hbase).
> 
> 
> 
> On Tue, Aug 13, 2013 at 2:50 PM, Jitendra Yadav <jeetuyadav200890@gmail.com> wrote:
> Hi,
>  
> One of your DN is marked as dead because NN is not able to get heartbeat message from
DN but NN still getting block information from dead node. This error is similar to a bug HDFS-1250
reported 2 years back and fixed in 0.20 release.
>  
> Can you please check the status of DN's in cluster.
>  
> #bin/hadoop dfsadmin -report
> 
> Thanks
> 
> On Tue, Aug 13, 2013 at 1:53 PM, Vimal Jain <vkjk89@gmail.com> wrote:
> Hi,
> I have configured Hadoop and Hbase in pseudo distributed mode.
> So far things were working fine , but suddenly i started receiving some exceptions in
my namenode and datanode log files.
> It keeps repeating and thus fills up my disk space.
> Please help here.
> 
> Exception in data node :-
> 
> 2013-07-31 19:39:51,094 WARN org.apache.hadoop.hdfs.server.datanode.DataNode: org.apache.hadoop.ipc.RemoteException:
java.io.IOException: Got blockRec
> eived message from unregistered or dead node blk_-4787262105551508952_28369
>     at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.blockReceived(FSNamesystem.java:4188)
>     at org.apache.hadoop.hdfs.server.namenode.NameNode.blockReceived(NameNode.java:1069)
>     at sun.reflect.GeneratedMethodAccessor56.invoke(Unknown Source)
>     at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>     at java.lang.reflect.Method.invoke(Method.java:597)
>     at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:578)
>     at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1393)
>     at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1389)
>     at java.security.AccessController.doPrivileged(Native Method)
>     at javax.security.auth.Subject.doAs(Subject.java:396)
>     at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1149)
>     at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1387)
> 
>     at org.apache.hadoop.ipc.Client.call(Client.java:1107)
>     at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:229)
>     at sun.proxy.$Proxy5.blockReceived(Unknown Source)
>     at org.apache.hadoop.hdfs.server.datanode.DataNode.offerService(DataNode.java:1006)
>     at org.apache.hadoop.hdfs.server.datanode.DataNode.run(DataNode.java:1527)
>     at java.lang.Thread.run(Thread.java:662)
> 
> Exception in name node :- 
> 
> 2013-07-31 19:39:50,671 WARN org.apache.hadoop.hdfs.StateChange: BLOCK* NameSystem.blockReceived:
blk_-4787262105551508952_28369 is received from dead
>  or unregistered node 192.168.20.30:50010
> 2013-07-31 19:39:50,671 ERROR org.apache.hadoop.security.UserGroupInformation: PriviledgedActionException
as:hadoop cause:java.io.IOException: Got blo
> ckReceived message from unregistered or dead node blk_-4787262105551508952_28369
> 2013-07-31 19:39:50,671 INFO org.apache.hadoop.ipc.Server: IPC Server handler 5 on 9000,
call blockReceived(DatanodeRegistration(192.168.20.30:50010, 
> storageID=DS-1816106352-192.168.20.30-50010-1369314076237, infoPort=50075, ipcPort=50020),
[Lorg.apache.hadoop.hdfs.protocol.Block;@64f2d559, [Ljava.l
> ang.String;@294f9d6) from 192.168.20.30:59764: error: java.io.IOException: Got blockReceived
message from unregistered or dead node blk_-4787262105551
> 508952_28369
> java.io.IOException: Got blockReceived message from unregistered or dead node blk_-4787262105551508952_28369
>     at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.blockReceived(FSNamesystem.java:4188)
>     at org.apache.hadoop.hdfs.server.namenode.NameNode.blockReceived(NameNode.java:1069)
>     at sun.reflect.GeneratedMethodAccessor56.invoke(Unknown Source)
>     at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>     at java.lang.reflect.Method.invoke(Method.java:597)
>     at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:578)
>     at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1393)
>     at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1389)
>     at java.security.AccessController.doPrivileged(Native Method)
>     at javax.security.auth.Subject.doAs(Subject.java:396)
>     at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1149)
>     at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1387)
> 
> 
> -- 
> Thanks and Regards,
> Vimal Jain
> 
> 
> 
> 
> -- 
> Thanks and Regards,
> Vimal Jain
> 
> 
> 
> -- 
> Thanks and Regards,
> Vimal Jain
> 
> 
> 
> -- 
> Thanks and Regards,
> Vimal Jain


Mime
View raw message