hadoop-hdfs-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Rakesh R <rake...@huawei.com>
Subject RE: HDFS multiple dfs_data_dir disbalance
Date Wed, 22 Oct 2014 13:31:34 GMT

Yes, there is a VolumeChoosingPolicy configuration "dfs.datanode.fsdataset.volume.choosing.policy"
in hdfs 
and by default it is configured to choose volumes in round-robin order. Hope you are using
the default policy?

Did you see any warnings or errors about the '/data/2' in datanode logs, datanode will do
sanity checks 
and if there is any exception it will skip this directory. Probably you can see this chance.


-----Original Message-----
From: Georgi Ivanov [mailto:ivanov@vesseltracker.com] 
Sent: 22 October 2014 17:17
To: user@hadoop.apache.org
Subject: HDFS multiple dfs_data_dir disbalance

My cluster is configured with 2 data dirs.

Usually hadoop is balancing the utilization of these dirs.
Now i have one node where /data/1 is 100% full and /data/2 is not.

Is there anything i can do about this, as this results in failed mapppers/rdecers ?


View raw message