hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Bryan Beaudreault <bbeaudrea...@hubspot.com>
Subject Re: Unbalanced region server
Date Tue, 24 Apr 2012 04:14:40 GMT
Unfortunately I don't have any experience with that version. You might try an hdfs rebalance,
but I'm not sure of the implications of that for hbase.  Hopefully someone else can be of
more help. Good luck!

Sent from iPhone.

On Apr 24, 2012, at 12:07 AM, David Charle <dbchar2012@gmail.com> wrote:

> On Mon, Apr 23, 2012 at 8:43 PM, Bryan Beaudreault <bbeaudreault@hubspot.com
>> wrote:
> 
>> Over time the balancer will move around both existing and new regions. A
>> few questions: How many tables do you have? Are they all configured
>> similarly? What version of hbase are you on?
>> 
>> 
> 70 tables.
> 
> hdfs://cluster1.com/hbase 2422000870355
> 
> Yes; all configured similarly; and have uniform conf across all regions.
> 
> 0.20.3 :) don't ask me to upgrade first, and then come back.. :(
> 
> balance_switch is not supported in this version.
> 
> Is there any other way to split and reclaim the space ?
> 
> Thanks for the help
> David
> 
> 
>> Sent from iPhone.
>> 
>> On Apr 23, 2012, at 11:39 PM, David Charle <dbchar2012@gmail.com> wrote:
>> 
>>> Does balancer_switch=true; rebalances any skew in the existing keys or it
>>> only does for all new regions ? The requests seems to be oK across all
>>> servers.
>>> 
>>> On Mon, Apr 23, 2012 at 8:03 PM, David Charle <dbchar2012@gmail.com>
>> wrote:
>>> 
>>>> Thanks Bryan.
>>>> 
>>>> Here is the regions count:
>>>> 
>>>> 1: 2616
>>>> 2: 2620
>>>> 3: 2623
>>>> 4: 2617
>>>> 5: 2617
>>>> 
>>>> The skew is in Node 2 where we have the space issue (double the size of
>>>> other 4 nodes).
>>>> 
>>>> 
>>>> On Mon, Apr 23, 2012 at 7:50 PM, Bryan Beaudreault <
>>>> bbeaudreault@hubspot.com> wrote:
>>>> 
>>>>> Do all nodes have the same number of regions?  If not you may not have
>> the
>>>>> balancer on.  You can turn it on using balancer_switch true in the
>> hbase
>>>>> shell.  The balancer also doesn't run if there is a region stuck in
>>>>> transition.  In this case if your data is growing rapidly I have seen
>>>>> regionservers become lopsided until I could clean up the stuck region
>> and
>>>>> re-enable the balancer.
>>>>> 
>>>>> Finally, depending on your version of hbase, the balancer only tries
to
>>>>> keep the number of regions similar across all regionservers.  If you
>> have
>>>>> tables of different max region sizes I could imagine a case where one
>>>>> region server unluckily is hosting more regions from the table with
>> larger
>>>>> region sizes.  This also might explain the imbalance in requests, if
>> one
>>>>> of
>>>>> your tables gets more traffic than other.  The HMaster UI would be
>> helpful
>>>>> for determining the spread of regions per table across regionservers.
>>>>> 
>>>>> On Mon, Apr 23, 2012 at 9:41 PM, David Charle <dbchar2012@gmail.com>
>>>>> wrote:
>>>>> 
>>>>>> Hi
>>>>>> 
>>>>>> What can make a region server unbalanced when it comes to space ?
(and
>>>>>> possibly requests too).
>>>>>> 
>>>>>> For example; I have 5 node cluster (replication factor of 3); and
in
>>>>> which
>>>>>> all 4 has same size; where as one node is almost double in size for
>>>>> /hbase.
>>>>>> 
>>>>>> Any help will be appreciated; esp what makes the skew and how to
fix
>> it
>>>>> ?
>>>>>> 
>>>>>> Thanks
>>>>>> David
>>>>>> 
>>>>> 
>>>> 
>>>> 
>> 

Mime
View raw message