Return-Path: X-Original-To: apmail-hadoop-user-archive@minotaur.apache.org Delivered-To: apmail-hadoop-user-archive@minotaur.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 6DB3B17D36 for ; Thu, 12 Feb 2015 06:35:18 +0000 (UTC) Received: (qmail 98861 invoked by uid 500); 12 Feb 2015 06:35:13 -0000 Delivered-To: apmail-hadoop-user-archive@hadoop.apache.org Received: (qmail 98733 invoked by uid 500); 12 Feb 2015 06:35:13 -0000 Mailing-List: contact user-help@hadoop.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@hadoop.apache.org Delivered-To: mailing list user@hadoop.apache.org Received: (qmail 98722 invoked by uid 99); 12 Feb 2015 06:35:13 -0000 Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 12 Feb 2015 06:35:13 +0000 X-ASF-Spam-Status: No, hits=1.5 required=5.0 tests=HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (athena.apache.org: domain of daemeonr@gmail.com designates 209.85.218.50 as permitted sender) Received: from [209.85.218.50] (HELO mail-oi0-f50.google.com) (209.85.218.50) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 12 Feb 2015 06:35:08 +0000 Received: by mail-oi0-f50.google.com with SMTP id v1so1576279oia.9 for ; Wed, 11 Feb 2015 22:32:33 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:from:date:message-id:subject:to :cc:content-type; bh=bxzaSo1qHDvNuDeRea6RMGYSekW+oSY4xZtH8gzGTOA=; b=owzhfLUtBqMDz7iZKJ1psqo6LsJFF+ji5ffs6B7WRZMfQif3jlOwix3xBCxXazOPpz xh68NS6+3z7T8+Js8U5ryEVhGSaqqHKOi0GzL3vULeRR5nSuTaMlw0Q/Fk/Bh42ywNzG 0YAM9QkZf4Du4eAEqR1z2loSwbN97Qrr/okKLwI5lcsfMDcXUHo6k+donwikrEJqYQYl dG/KOxaOTX0bBGlSokybYGvNT/ftqgD6gfzfihOTeWUSWofkzZ4cit9KvoqCLwo62+NF ILk3C5eOKBYKcxwHBES5giIDvnHiPNDRRgzDzGmk80EmALr86ZUsFXCkT4RPFZnvH7EX CaUg== X-Received: by 10.202.84.70 with SMTP id i67mr1396222oib.129.1423722753264; Wed, 11 Feb 2015 22:32:33 -0800 (PST) MIME-Version: 1.0 Received: by 10.182.128.234 with HTTP; Wed, 11 Feb 2015 22:32:03 -0800 (PST) In-Reply-To: References: <1230135568.3402592.1423690561779.JavaMail.yahoo@mail.yahoo.com> From: daemeon reiydelle Date: Wed, 11 Feb 2015 22:32:03 -0800 Message-ID: Subject: Re: hadoop cluster with non-uniform disk spec To: user@hadoop.apache.org Cc: Ravi Prakash Content-Type: multipart/alternative; boundary=001a113b0b90f79227050ede4894 X-Virus-Checked: Checked by ClamAV on apache.org --001a113b0b90f79227050ede4894 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable What have you set dfs.datanode.fsdataset.volume.choosing.policy to (assuming you are on a current version of Hadoop)? Is the policy set to org.apache.hadoop.hdfs.server.datanode.fsdataset.AvailableSpaceVolumeChoosi= ngPolicy? *.......* *=E2=80=9CLife should not be a journey to the grave with the intention of a= rriving safely in apretty and well preserved body, but rather to skid in broadside in a cloud of smoke,thoroughly used up, totally worn out, and loudly proclaiming =E2=80=9CWow! What a Ride!=E2=80=9D - Hunter ThompsonDaemeon C.= M. ReiydelleUSA (+1) 415.501.0198London (+44) (0) 20 8144 9872* On Wed, Feb 11, 2015 at 2:23 PM, Chen Song wrote: > Hey Ravi > > Here are my settings: > dfs.datanode.available-space-volume-choosing-policy.balanced-space-thresh= old =3D 21474836480 > (20G) > dfs.datanode.available-space-volume-choosing-policy.balanced-space-prefer= ence-fraction > =3D 0.85f > > Chen > > > On Wed, Feb 11, 2015 at 4:36 PM, Ravi Prakash wrote: > >> Hi Chen! >> >> Are you running the balancer? What are you setting dfs.datanode.availabl= e-space-volume-choosing-policy.balanced-space-threshold >> >> >> dfs.datanode.available-space-volume-choosing-policy.balanced-space-prefe= rence-fraction >> to? >> >> >> >> >> On Wednesday, February 11, 2015 7:44 AM, Chen Song < >> chen.song.82@gmail.com> wrote: >> >> >> We have a hadoop cluster consisting of 500 nodes. But the nodes are not >> uniform in term of disk spaces. Half of the racks are newer with 11 volu= mes >> of 1.1T on each node, while the other half have 5 volume of 900GB on eac= h >> node. >> >> dfs.datanode.fsdataset.volume.choosing.policy is set to >> org.apache.hadoop.hdfs.server.datanode.fsdataset.AvailableSpaceVolumeCho= osingPolicy. >> >> It winds up with the state of half of nodes are full while the other hal= f >> underutilized. I am wondering if there is a known solution for this prob= lem. >> >> Thank you for any suggestions. >> >> -- >> Chen Song >> >> >> >> > > > -- > Chen Song > > --001a113b0b90f79227050ede4894 Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: quoted-printable
What have you set dfs.datanode.fsdatas= et.volume.choosing.policy to (assuming you are on a current version of Hado= op)? Is the policy set to org.apache.hadoop.hdfs.server.datanode.fsdataset.= AvailableSpaceVolumeChoosingPolicy?



.......
=E2=80=9CLife should not be a journey to the grave with the intention o= f arriving safely in a
pretty and well preserved body, but rather to skid in broadside in a cloud of smoke,
thoroughly used up, totally worn out, and loudly proclaiming =E2=80=9CWow! What a Ride!=E2=80=9D
- = Hunter Thompson

Daemeon C.M. Reiydelle
USA (+1) 415.501.0198
L= ondon (+44) (0) 20 8144 9872

On Wed, Feb 11, 2015 at 2:23 PM, Chen Song <= span dir=3D"ltr"><chen.song.82@gmail.com> wrote:

On Wed, Feb 11, 2015 at 4:36 = PM, Ravi Prakash <ravihoo@ymail.com> wrote:
Hi Chen= !

dfs.datanode.available-space-volume-choosin= g-policy.balanced-space-preference-fraction
to?




On Wednesday, Feb= ruary 11, 2015 7:44 AM, Chen Song <chen.song.82@gmail.com> wrote:

We have a hadoop cluster consisti= ng of 500 nodes. But the nodes are not uniform in term of disk spaces. Half= of the racks are newer with 11 volumes of 1.1T on each node, while the oth= er half have 5 volume of 900GB on each node.

dfs.da= tanode.fsdataset.volume.choosing.policy is set to org.apache.hadoop.hdfs.se= rver.datanode.fsdataset.AvailableSpaceVolumeChoosingPolicy.
<= br>
It winds up with the state of half of nodes are full while th= e other half underutilized. I am wondering if there is a known solution for= this problem.

Thank you for any suggestions.

--
Chen Song



<= /div>



--
Chen Song


--001a113b0b90f79227050ede4894--