hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ted Dunning <tdunn...@veoh.com>
Subject Re: Balancer not balancing 100%?
Date Mon, 12 May 2008 17:30:30 GMT

I think the balancer has a pretty lenient feeling about what "balanced"
means.

If you want to shave off the last slivers, try the trick of increasing
replication on each file, one at a time and then decreasing it after 30-60
seconds.  You can do this at whatever rate your disk space limits you to
(i.e. If your disk is 80% full, you can double the replication on 1/4 of
your files without running out of disk).


On 5/11/08 11:48 AM, "Otis Gospodnetic" <otis_gospodnetic@yahoo.com> wrote:

> Oh, and on top of the above, I just observed that even though bin/hadoop
> balancer exits immediately and reports the cluster is fully balanced, I do see
> *very* few blocks (1-2 blocks per node) getting moved every time I run
> balancer.  It feels as if the balancer does actually find some blocks that it
> could move around, moves them, but then quickly gets lazy and just exits
> claiming the cluster is/was already balanced.  I just ran balancer about 10
> times and each time it moved a couple of blocks and then exited.
> 
> Makes me want to do ugly stuff like:
>     for ((i=1; i <= 9999; i++)); do echo $i; bin/hadoop balancer; done
> 
> 
> ...just to get to the point where all 4 nodes have the same number of blocks
> and thus the same percentage of disk used...
> 
> Otis
> --
> Sematext -- http://sematext.com/ -- Lucene - Solr - Nutch
> 
> 
> ----- Original Message ----
>> From: Otis Gospodnetic <otis_gospodnetic@yahoo.com>
>> To: core-user@hadoop.apache.org
>> Sent: Sunday, May 11, 2008 2:36:24 PM
>> Subject: Balancer not balancing 100%?
>> 
>> Hi,
>> 
>> I have 4 identical nodes in a Hadoop cluster (all functioning as DNs).  One
>> of 
>> the 4 nodes is a new node that I recently added.  I ran the balancer a few
>> times 
>> and it did move some of the blocks from the other 3 nodes to the new node.
>> However, the 4 nodes are still not 100% balanced (according to the GUI), even
>> though running bin/hadoop balancer says the cluster is balanced:
>> 
>> Time Stamp               Iteration#  Bytes Already Moved  Bytes Left To Move
>> Bytes Being Moved
>> The cluster is balanced. Exiting...
>> Balancing took 666.0 milliseconds
>> 
>> 
>> The 3 old DNs are about 60% full (around 24K blocks), which the 1 new DN is
>> only 
>> about 50% full (around 21K blocks).  I restarted the NN and re-ran the
>> balancer, 
>> bug got the same output: "The cluster is balanced. Exiting..."
>> 
>> Is this a bug or is it somehow possible for a cluster to be balanced, yet
>> have 
>> nodes with different number of blocks?
>> 
>> Thanks,
>> Otis
> 


Mime
View raw message