hadoop-mapreduce-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From sunww <spe...@outlook.com>
Subject write to most datanode fail quickly
Date Tue, 14 Oct 2014 09:31:58 GMT
Hi    I'm using hbase with about 20 regionserver. And  one regionserver failed to write  most
of datanodes quickly, finally cause this regionserver die. While other regionserver is ok.

logs like this:    java.io.IOException: Bad response ERROR for block BP-165080589-132.228.248.11-1371617709677:blk_5069077415583579127_39339217
from datanode 132.228.248.20:50010	at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer$ResponseProcessor.run(DFSOutputStream.java:681)2014-10-13
09:23:01,227 WARN org.apache.hadoop.hdfs.DFSClient: Error Recovery for block BP-165080589-132.228.248.11-1371617709677:blk_5069077415583579127_39339217
in pipeline 132.228.248.17:50010, 132.228.248.20:50010, 132.228.248.41:50010: bad datanode
132.228.248.20:500102014-10-13 09:23:32,021 WARN org.apache.hadoop.hdfs.DFSClient: DFSOutputStream
ResponseProcessor exception  for block BP-165080589-132.228.248.11-1371617709677:blk_5069077415583579127_39339415java.io.IOException:
Bad response ERROR for block BP-165080589-132.228.248.11-1371617709677:blk_5069077415583579127_39339415
from datanode 132.228.248.41:50010	at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer$ResponseProcessor.run(DFSOutputStream.java:681)
               then serveral  "firstBadLink error "    2014-10-13 09:23:33,390 INFO org.apache.hadoop.hdfs.DFSClient:
Exception in createBlockOutputStreamjava.io.IOException: Bad connect ack with firstBadLink
as 132.228.248.18:50010	at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.createBlockOutputStream(DFSOutputStream.java:1090)
           then serveral "Failed to add a datanode"    2014-10-13 09:23:44,331 WARN org.apache.hadoop.hdfs.DFSClient:
Error while syncingjava.io.IOException: Failed to add a datanode.  User may turn off this
feature by setting dfs.client.block.write.replace-datanode-on-failure.policy in configuration,
where the current policy is DEFAULT.  (Nodes: current=[132.228.248.17:50010, 132.228.248.35:50010],
original=[132.228.248.17:50010, 132.228.248.35:50010])
    the full log is in http://paste2.org/xfn16jm2        Any suggestion will be appreciated.
Thanks. 		 	   		  
Mime
View raw message