hadoop-mapreduce-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Mayank <mail2may...@gmail.com>
Subject Application errors with one disk on datanode getting filled up to 100%
Date Mon, 10 Jun 2013 09:36:09 GMT
We are running a hadoop cluster with 10 datanodes and a namenode. Each
datanode is setup with 4 disks (/data1, /data2, /data3, /data4), which each
disk having a capacity 414GB.

hdfs-site.xml has following property set:


        <description>Data dirs for DFS.</description>

Now we are facing a issue where in we find /data1 getting filled up quickly
and many a times we see it's usage running at 100% with just few megabytes
of free space. This issue is visible on 7 out of 10 datanodes at present.

We've some java applications which are writing to hdfs and many a times we
are seeing foloowing errors in our application logs:

java.io.IOException: All datanodes xxx.xxx.xxx.xxx:50010 are bad. Aborting...
	at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.processDatanodeError(DFSClient.java:3093)
	at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.access$2200(DFSClient.java:2586)
	at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:2790)

I went through some old discussions and looks like manual rebalancing is
what is required in this case and we should also have
dfs.datanode.du.reserved set up.

However I'd like to understand if this issue, with one disk getting filled
up to 100% can result into the issue which we are seeing in our

Also, are there any other peformance implications due to some of the disks
running at 100% usage on a datanode.
Mayank Joshi

Skype: mail2mayank
Mb.:  +91 8690625808

Blog: http://www.techynfreesouls.co.nr
PhotoStream: http://picasaweb.google.com/mail2mayank

Today is tommorrow I was so worried about yesterday ...

View raw message