hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "MauMau" <maumau...@gmail.com>
Subject Re: How does HBase perform load balancing?
Date Sun, 09 May 2010 01:34:08 GMT
Thanks, Ryan,

- To utilize the CPU and memory of new region servers:
  I do not have to do anything.
  The master automatically reassign existing regions to the new servers.
  (I'll search the docs for how soon the master performs reassignment when 
adding new servers)
- To utilize the storage space and I/O capacity of new region servers:
  Choose or combine the following:
  1. Automatic major compaction (once a day)
  2. Perform major compaction explicitly
  3. Use HDFS balancer

Maumau

----- Original Message ----- 
From: "Ryan Rawson" <ryanobjc@gmail.com>
To: <hbase-user@hadoop.apache.org>
Sent: Sunday, May 09, 2010 10:08 AM
Subject: Re: How does HBase perform load balancing?

> What I understood from the above is as follows. I'd appreciate if you 
> could
> point out if I am wrong.
>
> 1. I need to perform major-compaction to unassign regions from the 
> existing
> loaded region servers to a new region server.

This is not so - regions are automatically reassigned with no
compaction necessary.

> I cannot reassign the regions just by doing minor compaction and letting 
> the
> non-loaded new server perform major compaction later. Having the loaded
> existing server do heavy major compaction is a concern.

this is not what happens, regions are reassigned without requiring any
compaction of any kind.

> 2. "no rebalancing required" means that the blocks of HDFS files for 
> regions
> need not be moved from one datanode to another.

So when you add nodes to a new cluster, unless you are running the
HDFS balancer, data does not migrate.  As HBase naturally compacts
tables (once a day by default) it will end up rewriting data and
causing its migration. You can help accelerate this process by
manually kicking off a compaction for a large table if you have added
a lot of new machines.


Mime
View raw message