flume-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jeff Lord <jl...@cloudera.com>
Subject Re: java.io.IOException: Bad response ERROR for block ... from datanode
Date Wed, 07 Aug 2013 17:31:26 GMT

These errors usually indicate that there is a problem on your HDFS cluster.
You should probably investigate the health of the cluster first.


On Wed, Aug 7, 2013 at 7:21 AM, Miguel Coelho dos Santos
<miguel.coelho.santos@cern.ch> wrote:
> Hi,
> we are using flume to write data to hdfs.
> Our hdfs sinks recently started to report these errors.
> 07 Aug 2013 08:41:42,776 WARN  [ResponseProcessor for block BP-1897030109-WAS_IP_1-1343818418899:blk_-6113435038800423576_51499410]
(org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer$ResponseProcessor.run:747)  - DFSOutputStream
ResponseProcessor exception  for block BP-1897030109-WAS_IP_1-1343818418899:blk_-6113435038800423576_51499410
> java.io.IOException: Bad response ERROR for block BP-1897030109-WAS_IP_1-1343818418899:blk_-6113435038800423576_51499410
from datanode WAS_IP_2:50010
> 07 Aug 2013 08:41:42,776 WARN  [DataStreamer for file /tmp/syslog/px401/2013-07.tmp/29.tmp/FlumeData.1375857577304.tmp
block BP-1897030109-WAS_IP_1-1343818418899:blk_-6113435038800423576_51499410] (org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.setupPipelineForAppendOrRecovery:964)
 - Error Recovery for block BP-1897030109-WAS_IP_1-1343818418899:blk_-6113435038800423576_51499410
in pipeline, WAS_IP_3:50010, WAS_IP_2:50010: bad datanode WAS_IP_2:50010
> The errors are not referencing a single datanode, we see errors that reference all datanodes
in the hadoop cluster.
> Has anyone seen these errors in the past? What could be causing this?
> Happy to provide extra details about the setup or config files if needed.
> Please note that the various IPs were sanitized into WAS_IP_1, etc.
> Regards,
>  Miguel

View raw message