Return-Path: X-Original-To: apmail-hadoop-common-user-archive@www.apache.org Delivered-To: apmail-hadoop-common-user-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id CE6A31091C for ; Thu, 12 Feb 2015 11:56:07 +0000 (UTC) Received: (qmail 85346 invoked by uid 500); 12 Feb 2015 11:56:02 -0000 Delivered-To: apmail-hadoop-common-user-archive@hadoop.apache.org Received: (qmail 85224 invoked by uid 500); 12 Feb 2015 11:56:02 -0000 Mailing-List: contact user-help@hadoop.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@hadoop.apache.org Delivered-To: mailing list user@hadoop.apache.org Received: (qmail 85214 invoked by uid 99); 12 Feb 2015 11:56:02 -0000 Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 12 Feb 2015 11:56:02 +0000 X-ASF-Spam-Status: No, hits=-0.1 required=5.0 tests=HTML_MESSAGE,RCVD_IN_DNSWL_MED,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (nike.apache.org: domain of brahmareddy.battula@huawei.com designates 119.145.14.66 as permitted sender) Received: from [119.145.14.66] (HELO szxga03-in.huawei.com) (119.145.14.66) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 12 Feb 2015 11:55:36 +0000 Received: from 172.24.2.119 (EHLO SZXEML429-HUB.china.huawei.com) ([172.24.2.119]) by szxrg03-dlp.huawei.com (MOS 4.4.3-GA FastPath queued) with ESMTP id BBV88706; Thu, 12 Feb 2015 19:49:13 +0800 (CST) Received: from SZXEML510-MBX.china.huawei.com ([169.254.3.34]) by SZXEML429-HUB.china.huawei.com ([10.82.67.184]) with mapi id 14.03.0158.001; Thu, 12 Feb 2015 19:49:12 +0800 From: Brahma Reddy Battula To: "user@hadoop.apache.org" Subject: RE: hadoop cluster with non-uniform disk spec Thread-Topic: hadoop cluster with non-uniform disk spec Thread-Index: AQHQRhGiX+eGvDegxEeIanXLS98wpZzrczaAgAANUYCAAIhzgIAA3Mst Date: Thu, 12 Feb 2015 11:49:11 +0000 Message-ID: <8AD4EE147886274A8B495D6AF407DF698122B8F8@szxeml510-mbx.china.huawei.com> References: <1230135568.3402592.1423690561779.JavaMail.yahoo@mail.yahoo.com> , In-Reply-To: Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-originating-ip: [10.18.144.148] Content-Type: multipart/alternative; boundary="_000_8AD4EE147886274A8B495D6AF407DF698122B8F8szxeml510mbxchi_" MIME-Version: 1.0 X-CFilter-Loop: Reflected X-Mirapoint-Virus-RAPID-Raw: score=unknown(0), refid=str=0001.0A020202.54DC933A.0080,ss=1,re=0.000,recu=0.000,reip=0.000,cl=1,cld=1,fgs=0, ip=169.254.3.34, so=2013-05-26 15:14:31, dmn=2013-03-21 17:37:32 X-Mirapoint-Loop-Id: 59a55fda23c7cc6a14fc19e0ce9c2fc0 X-Virus-Checked: Checked by ClamAV on apache.org --_000_8AD4EE147886274A8B495D6AF407DF698122B8F8szxeml510mbxchi_ Content-Type: text/plain; charset="Windows-1252" Content-Transfer-Encoding: quoted-printable Hello daemeon reiydelle Is the policy set to org.apache.hadoop.hdfs.server.datanode.fsdataset.Avail= ableSpaceVolumeChoosingPolicy? >>Yes, you need to set this policy which will balance among the disks @Chen Song following settings controls what percentage of new block allocations will b= e sent to volumes with more available disk space than others dfs.datanode.available-space-volume-choosing-policy.balanced-space-threshol= d =3D 21474836480 (20G) dfs.datanode.available-space-volume-choosing-policy.balanced-space-preferen= ce-fraction =3D 0.85f Did you set while startup the cluster..? Thanks & Regards Brahma Reddy Battula ________________________________ From: daemeon reiydelle [daemeonr@gmail.com] Sent: Thursday, February 12, 2015 12:02 PM To: user@hadoop.apache.org Cc: Ravi Prakash Subject: Re: hadoop cluster with non-uniform disk spec What have you set dfs.datanode.fsdataset.volume.choosing.policy to (assumin= g you are on a current version of Hadoop)? Is the policy set to org.apache.= hadoop.hdfs.server.datanode.fsdataset.AvailableSpaceVolumeChoosingPolicy? ....... =93Life should not be a journey to the grave with the intention of arriving= safely in a pretty and well preserved body, but rather to skid in broadside in a cloud = of smoke, thoroughly used up, totally worn out, and loudly proclaiming =93Wow! What a= Ride!=94 - Hunter Thompson Daemeon C.M. Reiydelle USA (+1) 415.501.0198 London (+44) (0) 20 8144 9872 On Wed, Feb 11, 2015 at 2:23 PM, Chen Song > wrote: Hey Ravi Here are my settings: dfs.datanode.available-space-volume-choosing-policy.balanced-space-threshol= d =3D 21474836480 (20G) dfs.datanode.available-space-volume-choosing-policy.balanced-space-preferen= ce-fraction =3D 0.85f Chen On Wed, Feb 11, 2015 at 4:36 PM, Ravi Prakash > wrote: Hi Chen! Are you running the balancer? What are you setting dfs.datanode.available-s= pace-volume-choosing-policy.balanced-space-threshold dfs.datanode.available-space-volume-choosing-policy.balanced-space-preferen= ce-fraction to? On Wednesday, February 11, 2015 7:44 AM, Chen Song > wrote: We have a hadoop cluster consisting of 500 nodes. But the nodes are not uni= form in term of disk spaces. Half of the racks are newer with 11 volumes of= 1.1T on each node, while the other half have 5 volume of 900GB on each nod= e. dfs.datanode.fsdataset.volume.choosing.policy is set to org.apache.hadoop.h= dfs.server.datanode.fsdataset.AvailableSpaceVolumeChoosingPolicy. It winds up with the state of half of nodes are full while the other half u= nderutilized. I am wondering if there is a known solution for this problem. Thank you for any suggestions. -- Chen Song -- Chen Song --_000_8AD4EE147886274A8B495D6AF407DF698122B8F8szxeml510mbxchi_ Content-Type: text/html; charset="Windows-1252" Content-Transfer-Encoding: quoted-printable
Hello daemeon reiydelle=


Is the policy set to org.apache.hadoop.hdfs.server.datanode.fsdataset.Avai= lableSpaceVolumeChoosingPolicy?

>>Yes, you need to set this policy which will balance among the disks=

@Chen Song=

following settings controls what percentage of new block allocations will be sent to= volumes with more available disk space than others
dfs.datanode.available-space-volume-choosing-poli= cy.balanced-space-preference-fraction =3D 0.85f

Did you set while startup the cluster..?

Thanks & Regards

 Brahma Reddy Battula

 


From: daemeon reiydelle [daemeonr@gmail.c= om]
Sent: Thursday, February 12, 2015 12:02 PM
To: user@hadoop.apache.org
Cc: Ravi Prakash
Subject: Re: hadoop cluster with non-uniform disk spec

What have you set dfs.datanode.fsdataset.volume.choosing.policy to (assumin= g you are on a current version of Hadoop)? Is the policy set to org.apache.= hadoop.hdfs.server.datanode.fsdataset.AvailableSpaceVolumeChoosingPolicy?


.......
=93Life should not be a journey to the grave wi= th the intention of arriving safely in a
pretty and well preserved body, but rather to skid in broadside in a cloud = of smoke,
thoroughly used up, totally worn out, and loudly proclaiming =93Wow! What a= Ride!=94
- Hunter Thompson

Daemeon C.M. Reiydelle
USA (+1) 415.501.0198
London (+44) (0) 20 8144 9872=

On Wed, Feb 11, 2015 at 2:23 PM, Chen Song <chen.song.8= 2@gmail.com> wrote:

On Wed, Feb 11, 2015 at 4:36 PM, Ravi Prakash <ravihoo@ymail.co= m> wrote:
Hi Chen!

Are you running the balancer? What are you setting <= /span>dfs.datanode.availabl= e-space-volume-choosing-policy.balanced-space-threshold
dfs.datanode.avai= lable-space-volume-choosing-policy.balanced-space-preference-fraction
to?




On Wednesday, February 11, 2015 7:44 = AM, Chen Song <chen.song.82@gmail.com> wrote:


We have a hadoop cluster consisting of 500 nodes. But the = nodes are not uniform in term of disk spaces. Half of the racks are newer w= ith 11 volumes of 1.1T on each node, while the other half have 5 volume of = 900GB on each node.

dfs.datanode.fsdataset.volume.choosing.policy is set to org.apache.had= oop.hdfs.server.datanode.fsdataset.AvailableSpaceVolumeChoosingPolicy.

It winds up with the state of half of nodes are full while the other h= alf underutilized. I am wondering if there is a known solution for this pro= blem.

Thank you for any suggestions.

--
Chen Song






--
Chen Song


--_000_8AD4EE147886274A8B495D6AF407DF698122B8F8szxeml510mbxchi_--