hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Chen Song <chen.song...@gmail.com>
Subject Re: hadoop cluster with non-uniform disk spec
Date Wed, 11 Feb 2015 22:23:41 GMT
Hey Ravi

Here are my settings:
dfs.datanode.available-space-volume-choosing-policy.balanced-space-threshold
= 21474836480
(20G)
dfs.datanode.available-space-volume-choosing-policy.balanced-space-preference-fraction
= 0.85f

Chen


On Wed, Feb 11, 2015 at 4:36 PM, Ravi Prakash <ravihoo@ymail.com> wrote:

> Hi Chen!
>
> Are you running the balancer? What are you setting dfs.datanode.available-space-volume-choosing-policy.balanced-space-threshold
>
>
> dfs.datanode.available-space-volume-choosing-policy.balanced-space-preference-fraction
> to?
>
>
>
>
>   On Wednesday, February 11, 2015 7:44 AM, Chen Song <
> chen.song.82@gmail.com> wrote:
>
>
> We have a hadoop cluster consisting of 500 nodes. But the nodes are not
> uniform in term of disk spaces. Half of the racks are newer with 11 volumes
> of 1.1T on each node, while the other half have 5 volume of 900GB on each
> node.
>
> dfs.datanode.fsdataset.volume.choosing.policy is set to
> org.apache.hadoop.hdfs.server.datanode.fsdataset.AvailableSpaceVolumeChoosingPolicy.
>
> It winds up with the state of half of nodes are full while the other half
> underutilized. I am wondering if there is a known solution for this problem.
>
> Thank you for any suggestions.
>
> --
> Chen Song
>
>
>
>


-- 
Chen Song

Mime
View raw message