Mailing-List: contact core-user-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
Reply-To: core-user@hadoop.apache.org
Received-SPF: neutral (athena.apache.org: local policy)
DomainKey-Signature: a=rsa-sha1; s=serpent; d=yahoo-inc.com; c=nofws; q=dns;
	h=received:user-agent:date:subject:from:to:message-id:
	thread-topic:thread-index:in-reply-to:mime-version:content-type:
	content-transfer-encoding:return-path:x-originalarrivaltime;
	b=C916DN+Ub0vCsMivWLAj5zEWWfrLrun+ceMRMmACMfILOprinT6mJUCKfKXEPGJl
User-Agent: Microsoft-Entourage/11.3.6.070618
Date: Mon, 12 May 2008 10:53:16 -0700
Subject: Re: Balancer not balancing 100%?
From: Hairong Kuang <hairong@yahoo-inc.com>
To: <core-user@hadoop.apache.org>
Message-ID: <C44DD21C.4B45%hairong@yahoo-inc.com>
Thread-Topic: Balancer not balancing 100%?
Thread-Index: Aci0Vd/MHkHd4CBJEd2TiwAWy8rVfQAAy4yR
In-Reply-To: <C44DCCC6.3F0FA%tdunning@veoh.com>
Mime-version: 1.0
Content-type: text/plain;
	charset="US-ASCII"
Content-transfer-encoding: 7bit

Please check the balancer user guide at
http://issues.apache.org/jira/secure/attachment/12370966/BalancerUserGuide2.
pdf. As stated in the document, a cluster is balanced iff
 |utilization(DNi)-average utilization|<threshold
for each datanode DNi,

When you run a balancer, the default threshold is 10%. If you want a cluster
ends up to be more balanced, you may use a smaller threshold.

Good luck,
Hairong

On 5/12/08 10:30 AM, "Ted Dunning" <tdunning@veoh.com> wrote:

> 
> I think the balancer has a pretty lenient feeling about what "balanced"
> means.
> 
> If you want to shave off the last slivers, try the trick of increasing
> replication on each file, one at a time and then decreasing it after 30-60
> seconds.  You can do this at whatever rate your disk space limits you to
> (i.e. If your disk is 80% full, you can double the replication on 1/4 of
> your files without running out of disk).
> 
> 
> On 5/11/08 11:48 AM, "Otis Gospodnetic" <otis_gospodnetic@yahoo.com> wrote:
> 
>> Oh, and on top of the above, I just observed that even though bin/hadoop
>> balancer exits immediately and reports the cluster is fully balanced, I do
>> see
>> *very* few blocks (1-2 blocks per node) getting moved every time I run
>> balancer.  It feels as if the balancer does actually find some blocks that it
>> could move around, moves them, but then quickly gets lazy and just exits
>> claiming the cluster is/was already balanced.  I just ran balancer about 10
>> times and each time it moved a couple of blocks and then exited.
>> 
>> Makes me want to do ugly stuff like:
>>     for ((i=1; i <= 9999; i++)); do echo $i; bin/hadoop balancer; done
>> 
>> 
>> ...just to get to the point where all 4 nodes have the same number of blocks
>> and thus the same percentage of disk used...
>> 
>> Otis
>> --
>> Sematext -- http://sematext.com/ -- Lucene - Solr - Nutch
>> 
>> 
>> ----- Original Message ----
>>> From: Otis Gospodnetic <otis_gospodnetic@yahoo.com>
>>> To: core-user@hadoop.apache.org
>>> Sent: Sunday, May 11, 2008 2:36:24 PM
>>> Subject: Balancer not balancing 100%?
>>> 
>>> Hi,
>>> 
>>> I have 4 identical nodes in a Hadoop cluster (all functioning as DNs).  One
>>> of 
>>> the 4 nodes is a new node that I recently added.  I ran the balancer a few
>>> times 
>>> and it did move some of the blocks from the other 3 nodes to the new node.
>>> However, the 4 nodes are still not 100% balanced (according to the GUI),
>>> even
>>> though running bin/hadoop balancer says the cluster is balanced:
>>> 
>>> Time Stamp               Iteration#  Bytes Already Moved  Bytes Left To Move
>>> Bytes Being Moved
>>> The cluster is balanced. Exiting...
>>> Balancing took 666.0 milliseconds
>>> 
>>> 
>>> The 3 old DNs are about 60% full (around 24K blocks), which the 1 new DN is
>>> only 
>>> about 50% full (around 21K blocks).  I restarted the NN and re-ran the
>>> balancer, 
>>> bug got the same output: "The cluster is balanced. Exiting..."
>>> 
>>> Is this a bug or is it somehow possible for a cluster to be balanced, yet
>>> have 
>>> nodes with different number of blocks?
>>> 
>>> Thanks,
>>> Otis
>> 
>