hadoop-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Chen Song <chen.song...@gmail.com>
Subject hadoop cluster with non-uniform disk spec
Date Wed, 11 Feb 2015 15:44:00 GMT
We have a hadoop cluster consisting of 500 nodes. But the nodes are not
uniform in term of disk spaces. Half of the racks are newer with 11 volumes
of 1.1T on each node, while the other half have 5 volume of 900GB on each
node.

dfs.datanode.fsdataset.volume.choosing.policy is set to
org.apache.hadoop.hdfs.server.datanode.fsdataset.AvailableSpaceVolumeChoosingPolicy.

It winds up with the state of half of nodes are full while the other half
underutilized. I am wondering if there is a known solution for this problem.

Thank you for any suggestions.

-- 
Chen Song

Mime
View raw message