hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Arvind Sharma <arvind...@yahoo.com>
Subject DFSClient write error when DN down
Date Wed, 02 Dec 2009 16:02:39 GMT


I have seen similar error logs in the Hadoop Jira (Hadoop-2691, HDFS-795 ) but not sure this
one is exactly the same scenario.

Hadoop - 0.19.2

The client side DFSClient fails to write when few of the DN in a grid goes down.  I see this
error :

***************************

2009-11-13 13:45:27,815 WARN DFSClient | DFSOutputStream
ResponseProcessor exception for block
blk_3028932254678171367_1462691java.io.IOException: Bad response 1 for
block blk_30289322
54678171367_1462691 from datanode 10.201.9.225:50010
at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream$ResponseProcessor.run(DFSClient.java:2341)
2009-11-13 13:45:27,815 WARN             DFSClient |  Error Recovery for block blk_3028932254678171367_1462691
bad datanode[2] 10.201.9.225:50010 
2009-11-13 13:45:27,815 WARN DFSClient | Error Recovery for block
blk_3028932254678171367_1462691 in pipeline 10.201.9.218:50010,
10.201.9.220:50010, 10.201.9.225:50010: bad datanode 10
..201.9.225:50010 
2009-11-13 13:45:37,433 WARN DFSClient | DFSOutputStream
ResponseProcessor exception for block
blk_-6619123912237837733_1462799java.io.IOException: Bad response 1 for
block blk_-661912
3912237837733_1462799 from datanode 10.201.9.225:50010
at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream$ResponseProcessor.run(DFSClient.java:2341)2009-11-13
13:45:37,433 WARN             DFSClient |  Error Recovery for block blk_-6619123912237837733_1462799
bad datanode[1] 10.201.9.225:50010 
2009-11-13 13:45:37,433 WARN DFSClient | Error Recovery for block
blk_-6619123912237837733_1462799 in pipeline 10.201.9.218:50010,
10.201.9.225:50010: bad datanode 10.201.9.225:50010


***************************

The only way I could get my client program to write successfully to the DFS was to re-start
it.

Any suggestions how to get around this problem on the client side ?  As I understood, the
DFSClient APIs will take care of situations like this and the clients don't need to worry
about if some of the DN goes down.

Also, the replication factor is 3 in my setup and there are 10 DN (out of which TWO went down)


Thanks!
Arvind



      
Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message