Return-Path: X-Original-To: apmail-hadoop-user-archive@minotaur.apache.org Delivered-To: apmail-hadoop-user-archive@minotaur.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 1FE5517C08 for ; Thu, 12 Feb 2015 15:35:10 +0000 (UTC) Received: (qmail 5371 invoked by uid 500); 12 Feb 2015 15:35:04 -0000 Delivered-To: apmail-hadoop-user-archive@hadoop.apache.org Received: (qmail 5257 invoked by uid 500); 12 Feb 2015 15:35:04 -0000 Mailing-List: contact user-help@hadoop.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@hadoop.apache.org Delivered-To: mailing list user@hadoop.apache.org Received: (qmail 5242 invoked by uid 99); 12 Feb 2015 15:35:04 -0000 Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 12 Feb 2015 15:35:04 +0000 X-ASF-Spam-Status: No, hits=2.7 required=5.0 tests=FREEMAIL_ENVFROM_END_DIGIT,FREEMAIL_REPLY,HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (nike.apache.org: domain of chen.song.82@gmail.com designates 209.85.215.50 as permitted sender) Received: from [209.85.215.50] (HELO mail-la0-f50.google.com) (209.85.215.50) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 12 Feb 2015 15:34:39 +0000 Received: by lams18 with SMTP id s18so10790584lam.13 for ; Thu, 12 Feb 2015 07:33:07 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :content-type; bh=jJ3p8p6qXBBs+8xzZuaN+NBuKO8I8ABPRZjdMiSW4S4=; b=sXRu/Fng2fMdWLYkfgEW1ZDYw/aPJPtnJ7+af4Au1+yLNAaz0H/yQ+6TsStbb6As6H Kr4O53ceicN33iKqXAJMJOlzX2lEcyAEX5tWr49Ghjxb5uUzRZ1nfbOSrXvfo8T3TTAj eU4Z+SgaXS0isMi6ft1ACyU6xtCG9zy6uhpTYzYXwN2wfkAkdsSMx2LO7aiTFWS5mTNa H/M4pzSIo91IRCY26Pkyr18n9WbmFpDH+NqFXKlNqMrodOPsvhiCLs1j0qEhOo4kqysz o/9tJSt3HKj6ldPVgV9IAL4d/A7R7isl9wYigP8D6bueDNjR0B/NX5LHjjo7i+kUtF0E Zvgw== MIME-Version: 1.0 X-Received: by 10.152.87.3 with SMTP id t3mr2201601laz.19.1423755187668; Thu, 12 Feb 2015 07:33:07 -0800 (PST) Received: by 10.25.91.143 with HTTP; Thu, 12 Feb 2015 07:33:07 -0800 (PST) In-Reply-To: <8AD4EE147886274A8B495D6AF407DF698122B8F8@szxeml510-mbx.china.huawei.com> References: <1230135568.3402592.1423690561779.JavaMail.yahoo@mail.yahoo.com> <8AD4EE147886274A8B495D6AF407DF698122B8F8@szxeml510-mbx.china.huawei.com> Date: Thu, 12 Feb 2015 10:33:07 -0500 Message-ID: Subject: Re: hadoop cluster with non-uniform disk spec From: Chen Song To: user@hadoop.apache.org Content-Type: multipart/alternative; boundary=001a11c35576354d76050ee5d678 X-Virus-Checked: Checked by ClamAV on apache.org --001a11c35576354d76050ee5d678 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable *@Leo Leung* Yes, dfs.datanode.data.dir is set correctly. @Brahma Reddy Battula Initially all the nodes we had were 5-disk nodes. Then we added a few racks of 11-disk nodes. We are using CDH distribution and we set these settings when we upgraded from CDH4 to CDH5. To make it more clear, at this moment, all nodes (regardless of 5 disks or 11 disks) have roughly the same number of blocks, thus the same amount of data stored. It seems data blocks are evenly distributed to the nodes regardless of whether it is a 5-disk or 11-disk node. Is this expected behavior? The concern is that as more data coming in, the 5-disk nodes are reaching to its configured capacity, while 11-disk nodes why below its capacity, because the latter have more space collectively on each node. I don't know if it is expected or my concern is valid? Chen On Thu, Feb 12, 2015 at 6:49 AM, Brahma Reddy Battula < brahmareddy.battula@huawei.com> wrote: > Hello daemeon reiydelle > > > Is the policy set to > org.apache.hadoop.hdfs.server.datanode.fsdataset.AvailableSpaceVolumeChoo= singPolicy? > > >>Yes, you need to set this policy which will balance among the disks > > *@Chen Song* > > following settings controls what percentage of new block allocations will > be sent to volumes with more available disk space than others > > dfs.datanode.available-space-volume-choosing-policy.balanced-space-thres= hold =3D 21474836480 > (20G) > dfs.datanode.available-space-volume-choosing-policy.balanced-space-prefe= rence-fraction > =3D 0.85f > > > Did you set while startup the cluster..? > > > Thanks & Regards > > Brahma Reddy Battula > > > > > ------------------------------ > *From:* daemeon reiydelle [daemeonr@gmail.com] > *Sent:* Thursday, February 12, 2015 12:02 PM > *To:* user@hadoop.apache.org > *Cc:* Ravi Prakash > *Subject:* Re: hadoop cluster with non-uniform disk spec > > What have you set dfs.datanode.fsdataset.volume.choosing.policy to > (assuming you are on a current version of Hadoop)? Is the policy set to > org.apache.hadoop.hdfs.server.datanode.fsdataset.AvailableSpaceVolumeChoo= singPolicy? > > > > > * ....... * > > > > > > > *=E2=80=9CLife should not be a journey to the grave with the intention of= arriving > safely in a pretty and well preserved body, but rather to skid in broadsi= de > in a cloud of smoke, thoroughly used up, totally worn out, and loudly > proclaiming =E2=80=9CWow! What a Ride!=E2=80=9D - Hunter Thompson Daemeon= C.M. Reiydelle > USA (+1) 415.501.0198 <%28%2B1%29%20415.501.0198> London (+44) (0) 20 814= 4 > 9872 <%28%2B44%29%20%280%29%2020%208144%209872>* > > On Wed, Feb 11, 2015 at 2:23 PM, Chen Song wrote= : > >> Hey Ravi >> >> Here are my settings: >> dfs.datanode.available-space-volume-choosing-policy.balanced-space-thres= hold =3D 21474836480 >> (20G) >> dfs.datanode.available-space-volume-choosing-policy.balanced-space-pref= erence-fraction >> =3D 0.85f >> >> Chen >> >> >> On Wed, Feb 11, 2015 at 4:36 PM, Ravi Prakash wrote: >> >>> Hi Chen! >>> >>> Are you running the balancer? What are you setting dfs.datanode.availa= ble-space-volume-choosing-policy.balanced-space-threshold >>> >>> >>> dfs.datanode.available-space-volume-choosing-policy.balanced-space-pref= erence-fraction >>> to? >>> >>> >>> >>> >>> On Wednesday, February 11, 2015 7:44 AM, Chen Song < >>> chen.song.82@gmail.com> wrote: >>> >>> >>> We have a hadoop cluster consisting of 500 nodes. But the nodes are >>> not uniform in term of disk spaces. Half of the racks are newer with 11 >>> volumes of 1.1T on each node, while the other half have 5 volume of 900= GB >>> on each node. >>> >>> dfs.datanode.fsdataset.volume.choosing.policy is set to >>> org.apache.hadoop.hdfs.server.datanode.fsdataset.AvailableSpaceVolumeCh= oosingPolicy. >>> >>> It winds up with the state of half of nodes are full while the other >>> half underutilized. I am wondering if there is a known solution for thi= s >>> problem. >>> >>> Thank you for any suggestions. >>> >>> -- >>> Chen Song >>> >>> >>> >>> >> >> >> -- >> Chen Song >> >> > --=20 Chen Song --001a11c35576354d76050ee5d678 Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: quoted-printable
@Leo Leung
Yes,=C2=A0dfs.datanode.data.dir is set correctly.

=

@Brahma Reddy Battula


Initi= ally all the nodes we had were 5-disk nodes. Then we added a few racks of 1= 1-disk nodes. We are using CDH distribution and we set these settings when = we upgraded from CDH4 to CDH5.

To make it more cle= ar, at this moment, all nodes (regardless of 5 disks or 11 disks) have roug= hly the same number of blocks, thus the same amount of data stored. It seem= s data blocks are evenly distributed to the nodes regardless of whether it = is a 5-disk or 11-disk node. Is this expected behavior?

The concern is that as more data coming in, the 5-disk nodes are reac= hing to its configured capacity, while 11-disk nodes why below its capacity= , because the latter have more space collectively on each node.
<= br>
I don't know if it is expected or my concern is valid?

Chen


On Thu, Feb 12, 2015 at 6:49 AM, Brahm= a Reddy Battula <brahmareddy.battula@huawei.com> wrote:
Hello daemeon reiydelle


Is the policy set to org.apache.hadoop.hdfs.server.datano= de.fsdataset.AvailableSpaceVolumeChoosingPolicy?

>>Yes, you need to set this policy which will balance among the disks=

@Chen Song=

following settings controls what percentage of new block allocations will be sent to= volumes with more available disk space than others
Did you set while startup the cluster..?

Thanks & Regards

=C2=A0Brahma Reddy Battula

=C2=A0


Fro= m: daemeon reiydelle [daemeonr@gmail.com]
Sent: Thursday, February 12, 2015 12:02 PM
To: user= @hadoop.apache.org
Cc: Ravi Prakash
Subject: Re: hadoop cluster with non-uniform disk spec

What have you set dfs.datanode.fsdataset.volume.choosing.policy to (assumin= g you are on a current version of Hadoop)? Is the policy set to org.apache.= hadoop.hdfs.server.datanode.fsdataset.AvailableSpaceVolumeChoosingPolicy?


.......
=E2=80=9CLife should not be a journey to the gr= ave with the intention of arriving safely in a
pretty and well preserved body, but rather to skid in broadside in a cloud = of smoke,
thoroughly used up, totally worn out, and loudly proclaiming =E2=80=9CWow! = What a Ride!=E2=80=9D
- Hunter Thompson

Daemeon C.M. Reiydelle
USA (+1) 415.501.0198
London (+44) (0) 20 8144 9872=

On Wed, Feb 11, 2015 at 2:23 PM, Chen Song <chen.song.8= 2@gmail.com> wrote:
Hey Ravi

Here are my settings:

On Wed, Feb 11, 2015 at 4:36 PM, Ravi Prakash <ravihoo@ymail.co= m> wrote:
Hi Chen!

Are you running the balancer? What are you setting <= /span>dfs.= datanode.available-space-volume-choosing-policy.balanced-space-threshold
= dfs.datanode.available-space-volume-choosing-policy.balanced-space-preferen= ce-fraction
to?




On Wednesday, February 11, 2015 7:44 = AM, Chen Song <chen.song.82@gmail.com> wrote:


We have a hadoop cluster consisting of 500 nodes. But the = nodes are not uniform in term of disk spaces. Half of the racks are newer w= ith 11 volumes of 1.1T on each node, while the other half have 5 volume of = 900GB on each node.

dfs.datanode.fsdataset.volume.choosing.policy is set to org.apache.had= oop.hdfs.server.datanode.fsdataset.AvailableSpaceVolumeChoosingPolicy.

It winds up with the state of half of nodes are full while the other h= alf underutilized. I am wondering if there is a known solution for this pro= blem.

Thank you for any suggestions.

--
Chen Song






--
Chen Song





--
Chen Song

--001a11c35576354d76050ee5d678--