hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Andrew Purtell <apurt...@apache.org>
Subject Re: Hbase cluster configuration
Date Tue, 03 Feb 2009 23:03:10 GMT
Hi Michael,

I have found that trial and error is necessary now. There are no
clear formulas. How large the system can scale depends entirely
on the distributions of various aspects of your data set and on
the application specific load.

Cluster start up is the most demanding time. If you have an
inadequate number of xceivers available in the data node, you
will see regions fail to deploy at start up with transient
errors recorded in the master log regarding missing blocks. 
This is an indication you need to increase data node resources.
I keep a tail of the master log up in a window when the cluster
is starting up. Add a grep on "ERROR" if you just want to catch
exceptional conditions. When you see this, increase the number
of configured xceivers by a factor of two and restart. 

You should also increase the number of handlers in the data 
node configuration to the number of nodes in your cluster. 

Hope this helps, 

   - Andy

> From: Michael Dagaev
> Do you know the upper bound for regions per node for 0.18?
> As I understand, 1000 regions is still OK but 4000 is not.
> How can I estimate the amount of memory and number of
> xceivers for 0.18 if I know the key and value size ?


View raw message