hadoop-hdfs-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ravi Prakash <ravi...@ymail.com>
Subject Re: hadoop cluster with non-uniform disk spec
Date Wed, 11 Feb 2015 21:36:01 GMT
Hi Chen!
Are you running the balancer? What are you setting dfs.datanode.available-space-volume-choosing-policy.balanced-space-threshold



     On Wednesday, February 11, 2015 7:44 AM, Chen Song <chen.song.82@gmail.com> wrote:

 We have a hadoop cluster consisting of 500 nodes. But the nodes are not uniform in term of
disk spaces. Half of the racks are newer with 11 volumes of 1.1T on each node, while the other
half have 5 volume of 900GB on each node.
dfs.datanode.fsdataset.volume.choosing.policy is set to org.apache.hadoop.hdfs.server.datanode.fsdataset.AvailableSpaceVolumeChoosingPolicy.

It winds up with the state of half of nodes are full while the other half underutilized. I
am wondering if there is a known solution for this problem.
Thank you for any suggestions.

Chen Song

View raw message