hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Atul Aggarwal <aagga...@ncsu.edu>
Subject Any hint for DataXceiver error with HDFS (Invalid Blocks)?
Date Sat, 22 Feb 2014 19:51:38 GMT
I have a small hadoop cluster 8 nodes running native hadoop 1.0.2. NN and
SNN on different nodes. Nodes themselves are beefed up nodes with 20+ Gig
of memory. I am seeing the DataCeiver error repeatedly in my Datanodes
logs. (Usig with hive and pig)

I know it may be because of the setting in HDFS-Site.xml and I set it to
4096 already as below:

  <property><name>dfs.datanode.max.xcievers</name><value>4096</value></property>

Also, I have set the ulimit to be same as above for HDFS user. Still I get
the error as

 2014-02-22 00:40:36,021 WARN
org.apache.hadoop.hdfs.server.datanode.DataNode:
DatanodeRegistration(10.22.96.9:50010,
storageID=DS-1389939194-10.22.96.9-50010-1345070063427,
infoPort=50075,\
 ipcPort=50020):Got exception while serving
blk_-1430839469926724904_1952628 to /10.22.96.9:
java.io.IOException: Block blk_-1430839469926724904_1952628 is not valid.
        at org.apache.hadoop.hdfs.server.datanode.FSDataset.getBlockFile(FSDataset.java:1072)
        at org.apache.hadoop.hdfs.server.datanode.FSDataset.getLength(FSDataset.java:1035)
        at org.apache.hadoop.hdfs.server.datanode.FSDataset.getVisibleLength(FSDataset.java:1045)
        at org.apache.hadoop.hdfs.server.datanode.BlockSender.<init>(BlockSender.java:94)
        at org.apache.hadoop.hdfs.server.datanode.DataXceiver.readBlock(DataXceiver.java:189)
        at org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:99)
        at java.lang.Thread.run(Thread.java:662)

 2014-02-22 00:40:36,021 ERROR
org.apache.hadoop.hdfs.server.datanode.DataNode:
DatanodeRegistration(10.22.96.9:50010,
storageID=DS-1389939194-10.22.96.9-50010-1345070063427,
infoPort=50075\
, ipcPort=50020):DataXceiver
java.io.IOException: Block blk_-1430839469926724904_1952628 is not valid.
        at org.apache.hadoop.hdfs.server.datanode.FSDataset.getBlockFile(FSDataset.java:1072)
        at org.apache.hadoop.hdfs.server.datanode.FSDataset.getLength(FSDataset.java:1035)
        at org.apache.hadoop.hdfs.server.datanode.FSDataset.getVisibleLength(FSDataset.java:1045)
        at org.apache.hadoop.hdfs.server.datanode.BlockSender.<init>(BlockSender.java:94)
        at org.apache.hadoop.hdfs.server.datanode.DataXceiver.readBlock(DataXceiver.java:189)
        at org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:99)
        at java.lang.Thread.run(Thread.java:662)

Any hints or explanation of the this happening again and again will be
really helpful for my better understanding for the issue. Or how can I
debug this issue to resolve this issue at root or atleast be able to
explain it to someone who asks :)

Thanks for looking at it,

 -Atul

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message