hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Brahma Reddy Battula <brahmareddy.batt...@huawei.com>
Subject RE: hadoop cluster with non-uniform disk spec
Date Thu, 12 Feb 2015 11:49:11 GMT
Hello daemeon reiydelle

Is the policy set to org.apache.hadoop.hdfs.server.datanode.fsdataset.AvailableSpaceVolumeChoosingPolicy?

>>Yes, you need to set this policy which will balance among the disks

@Chen Song

following settings controls what percentage of new block allocations will be sent to volumes
with more available disk space than others

dfs.datanode.available-space-volume-choosing-policy.balanced-space-threshold = 21474836480
dfs.datanode.available-space-volume-choosing-policy.balanced-space-preference-fraction = 0.85f

Did you set while startup the cluster..?

Thanks & Regards

 Brahma Reddy Battula

From: daemeon reiydelle [daemeonr@gmail.com]
Sent: Thursday, February 12, 2015 12:02 PM
To: user@hadoop.apache.org
Cc: Ravi Prakash
Subject: Re: hadoop cluster with non-uniform disk spec

What have you set dfs.datanode.fsdataset.volume.choosing.policy to (assuming you are on a
current version of Hadoop)? Is the policy set to org.apache.hadoop.hdfs.server.datanode.fsdataset.AvailableSpaceVolumeChoosingPolicy?

“Life should not be a journey to the grave with the intention of arriving safely in a
pretty and well preserved body, but rather to skid in broadside in a cloud of smoke,
thoroughly used up, totally worn out, and loudly proclaiming “Wow! What a Ride!”
- Hunter Thompson

Daemeon C.M. Reiydelle
USA (+1) 415.501.0198
London (+44) (0) 20 8144 9872

On Wed, Feb 11, 2015 at 2:23 PM, Chen Song <chen.song.82@gmail.com<mailto:chen.song.82@gmail.com>>
Hey Ravi

Here are my settings:
dfs.datanode.available-space-volume-choosing-policy.balanced-space-threshold = 21474836480
dfs.datanode.available-space-volume-choosing-policy.balanced-space-preference-fraction = 0.85f


On Wed, Feb 11, 2015 at 4:36 PM, Ravi Prakash <ravihoo@ymail.com<mailto:ravihoo@ymail.com>>
Hi Chen!

Are you running the balancer? What are you setting dfs.datanode.available-space-volume-choosing-policy.balanced-space-threshold

On Wednesday, February 11, 2015 7:44 AM, Chen Song <chen.song.82@gmail.com<mailto:chen.song.82@gmail.com>>

We have a hadoop cluster consisting of 500 nodes. But the nodes are not uniform in term of
disk spaces. Half of the racks are newer with 11 volumes of 1.1T on each node, while the other
half have 5 volume of 900GB on each node.

dfs.datanode.fsdataset.volume.choosing.policy is set to org.apache.hadoop.hdfs.server.datanode.fsdataset.AvailableSpaceVolumeChoosingPolicy.

It winds up with the state of half of nodes are full while the other half underutilized. I
am wondering if there is a known solution for this problem.

Thank you for any suggestions.

Chen Song

Chen Song

View raw message