hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From stack <st...@duboce.net>
Subject Re: Multiple start regions in a table
Date Wed, 12 Nov 2008 20:14:47 GMT
Arthur van Hoff wrote:
> We are using hbase 0.18.1. The count seems happy to count the first
> 100,000 rows...
> I was able to merge the two regions.
>
> However, I noticed that there are many overlapping regions in this
> table. Is that expected?
>   
No.  Only time you should see overlap is if the first region in list of 
regions out of .META. is offline and parent of the next two regions that 
follow.

> Here are some of the start/end keys:
>
>                     -0000010225 (newly merged)
> 0000003046-0000005771 (overlaps with the previous section)
> 0000005771-0000008992
> 0000008992-0000012909
> 0000010225-0000020097 (overlaps with the previous section)
> 0000012909-0000015835 (overlaps with the previous section)
>   

Do your MR jobs work now?

> ...
>
> How can this happen?
>   

Should never happen.

Could happen if edits to .META. were dropped.  Have seen rare case of 
this split events but the gaping holes should be closed at least in the 
version you are running.

If you can't have your MR run to completion, and if overlapping region 
is not offlined, may have to do more repairs.  Then, run with DEBUG 
enabled and if it happens again, should be able to see by examination of 
logs how it came about.

St.Ack


> On Wed, Nov 12, 2008 at 11:05 AM, stack <stack@duboce.net> wrote:
>   
>> Arthur van Hoff wrote:
>>     
>>> Below are the two regions in the .META. scan. I don't think either of
>>> them is disabled.
>>>
>>>       
>> Right.  One should be offlined.  Which version of hbase is this?
>>
>>     
>>> The table is too large to count, and I am not sure how to enable DEBUG.
>>>
>>>
>>>       
>> Well, looks like problem is in first region so you could start the count and
>> looks like issue should pop up quick.
>>
>> To enable DEBUG, http://wiki.apache.org/hadoop/Hbase/FAQ#5.
>>
>> To 'repair', your table, shut it down and then run:
>>
>> $ ./bin/hbase org.apache.hadoop.hbase.util.Merge
>> Usage: bin/hbase merge <table-name> <region-1> <region-2>
>>
>> Supply table name, and two regions to merge.
>>
>> Let us know how it goes.
>>
>> St.Ack
>>
>>     
>>>  wikipedia,,1226039865543    column=historian:assignment,
>>> timestamp=1226471612375, value=Region assigned to server
>>> 192.168.33.31:60020
>>>  wikipedia,,1226039865543    column=historian:open,
>>> timestamp=1226471627434, value=Region opened on server :
>>> gore.int.ellerdale.com
>>>  wikipedia,,1226039865543    column=historian:split,
>>> timestamp=1226039870630, value=Region split from  :
>>> wikipedia,,1226039822665
>>>  wikipedia,,1226039865543    column=info:regioninfo,
>>> timestamp=1226039870633, value=REGION => {NAME =>
>>> 'wikipedia,,1226039865543', STARTKEY => '', ENDKEY => '0000020097',
>>> ENCODED => 1423314116, TABLE => {{NAME => 'wikipedia', IS_ROOT =>
>>> 'false', IS_META => 'false', FAMILIES => [{NAME => 'data', BLOOMFILTER
>>> => 'false', VERSIONS => '3', COMPRESSION => 'NONE', LENGTH =>
>>> '2147483647', TTL => '-1', IN_MEMORY => 'false', BLOCKCACHE =>
>>> 'false'}]}}
>>>  wikipedia,,1226039865543    column=info:server,
>>> timestamp=1226471627433, value=192.168.33.31:60020
>>>  wikipedia,,1226039865543    column=info:serverstartcode,
>>> timestamp=1226471627433, value=1226471577772
>>>  wikipedia,,1226469112210    column=historian:assignment,
>>> timestamp=1226471612379, value=Region assigned to server
>>> 192.168.33.31:60020
>>>  wikipedia,,1226469112210    column=historian:open,
>>> timestamp=1226471630431, value=Region opened on server :
>>> gore.int.ellerdale.com
>>>  wikipedia,,1226469112210    column=historian:split,
>>> timestamp=1226469113121, value=Region split from  :
>>> wikipedia,,1226356665025
>>>  wikipedia,,1226469112210    column=info:regioninfo,
>>> timestamp=1226469113125, value=REGION => {NAME =>
>>> 'wikipedia,,1226469112210', STARTKEY => '', ENDKEY => '0000003046',
>>> ENCODED => 566585206, TABLE => {{NAME => 'wikipedia', IS_ROOT =>
>>> 'false', IS_META => 'false', FAMILIES => [{NAME => 'data', BLOOMFILTER
>>> => 'false', VERSIONS => '3', COMPRESSION => 'NONE', LENGTH =>
>>> '2147483647', TTL => '-1', IN_MEMORY => 'false', BLOCKCACHE =>
>>> 'false'}]}}
>>>  wikipedia,,1226469112210    column=info:server,
>>> timestamp=1226471630430, value=192.168.33.31:60020
>>>  wikipedia,,1226469112210    column=info:serverstartcode,
>>> timestamp=1226471630430, value=1226471577772
>>>
>>>
>>> On Wed, Nov 12, 2008 at 10:09 AM, stack <stack@duboce.net> wrote:
>>>
>>>       
>>>> Are they both enabled?
>>>>
>>>> You can see by doing "scan '.META.'" in the shell and then look at the
>>>> particular regions in the output.  Disabled regions will have a disabled
>>>> attribute.
>>>>
>>>> What error are you seeing when in your job?
>>>>
>>>> You could enable DEBUG and then in shell try running "count 'TABLENAME'".
>>>>  With DEBUG enabled, you'll see the regions loading and you'll be able to
>>>> pinpoint the problematic one.  Or, if your table is large, run a "scan
>>>> 'TABLENAME', ['NONEXISTANT_COLUMNNAME']".  That'll cause the scanner to spin
>>>> through all regions quickly -- there'll be no values found as it scans --
>>>> and again with DEBUG you can see the regions loading and figure the bad one.
>>>>
>>>> When you've found the bad region, look at its listing in the '.META.'
>>>> output.  Or paste it in here and we'll take a look for you.
>>>>
>>>> Thanks Arthur,
>>>> St.Ack
>>>>
>>>>
>>>> Arthur van Hoff wrote:
>>>>
>>>>         
>>>>> Hi,
>>>>>
>>>>> I have a table which appears to have two regions without a start key.
>>>>> Is that expected? My job is getting stuck scanning these two regions.
>>>>>
>>>>> Any suggestions?
>>>>>
>>>>> Thanks.
>>>>>
>>>>>
>>>>>           
>>>
>>> --
>>> Arthur van Hoff - Grand Master of Alphabetical Order
>>> The Ellerdale Project, Menlo Park, CA
>>> avh@ellerdale.com, 650-283-0842
>>>
>>>       
>>     
>
>
>
>   


Mime
View raw message