hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ryan Rawson <ryano...@gmail.com>
Subject Re: How does HBase perform load balancing?
Date Sun, 09 May 2010 01:36:05 GMT
I dont think the docs say anything about how the balancer works, but
it is immediate. As soon as a Regionserver makes the 'report for duty'
call back to the master the master will begin reassigning regions. It
can take a few for regions to flush then close and move, but it is
done fairly efficiently and quickly.

The other points are quite accurate - thanks for the summary.  If you
are interested in writing blog articles or documentation, we would
greatly appreciate it, and can link/embed it on our site/docs. There
are only so many hours in the day alas.

Thanks again,
-ryan

On Sat, May 8, 2010 at 6:34 PM, MauMau <maumau307@gmail.com> wrote:
> Thanks, Ryan,
>
> - To utilize the CPU and memory of new region servers:
>  I do not have to do anything.
>  The master automatically reassign existing regions to the new servers.
>  (I'll search the docs for how soon the master performs reassignment when
> adding new servers)
> - To utilize the storage space and I/O capacity of new region servers:
>  Choose or combine the following:
>  1. Automatic major compaction (once a day)
>  2. Perform major compaction explicitly
>  3. Use HDFS balancer
>
> Maumau
>
> ----- Original Message ----- From: "Ryan Rawson" <ryanobjc@gmail.com>
> To: <hbase-user@hadoop.apache.org>
> Sent: Sunday, May 09, 2010 10:08 AM
> Subject: Re: How does HBase perform load balancing?
>
>> What I understood from the above is as follows. I'd appreciate if you
>> could
>> point out if I am wrong.
>>
>> 1. I need to perform major-compaction to unassign regions from the
>> existing
>> loaded region servers to a new region server.
>
> This is not so - regions are automatically reassigned with no
> compaction necessary.
>
>> I cannot reassign the regions just by doing minor compaction and letting
>> the
>> non-loaded new server perform major compaction later. Having the loaded
>> existing server do heavy major compaction is a concern.
>
> this is not what happens, regions are reassigned without requiring any
> compaction of any kind.
>
>> 2. "no rebalancing required" means that the blocks of HDFS files for
>> regions
>> need not be moved from one datanode to another.
>
> So when you add nodes to a new cluster, unless you are running the
> HDFS balancer, data does not migrate.  As HBase naturally compacts
> tables (once a day by default) it will end up rewriting data and
> causing its migration. You can help accelerate this process by
> manually kicking off a compaction for a large table if you have added
> a lot of new machines.
>
>

Mime
View raw message