hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Adam Phelps <...@opendns.com>
Subject Re: Hbck errors
Date Tue, 22 Mar 2011 02:19:48 GMT
On 3/6/11 10:08 AM, Marc Limotte wrote:
>     1. split started on region A
>     2. region A was offlined
>     3. The daughter regions were created in HDFS with the reference files
>     4. .META. was updated for region A
>     5. **** server crashed
>
> So, the new daughter entries were never added to .META.
>
> We first tried to online region A with the shell command "assign'.  Figuring
> that hbase would just find and split region A again.  This seemed to have no
> effect... not sure why, maybe because region A already had splitA/B
> entries?  Region A remained offline.  We also tried to force it to split
> region A, using the shell command "split".  Again no effect.
>
> Finally we tried to manually complete the split that had started.  Peter
> manually inserted the two daughter regions into .META.  We then tried to
> force a compact from the shell, this failed with a NSRE.  So we onlined
> region A with the "assign"  command-- it worked this time.  And now we seem
> to be up again, compact works, data loads work, hbck checks out!

It looks like we've come up against a problem that looks identical to 
the one you described.  How did you go about manually inserting the two 
child regions?

Our current thought on fixing it is to use the hbase shell to remove the 
entries for the child regions and rewrite the region's entry such that 
OFFLINE => false and SPLIT => false (ie both currently true) but we're 
not sure if thats a good solution.

*** .META. info for the problem region ***

  domains,1932334:2011/02/18/03:com.photobucket.i654,1 
column=info:regioninfo, timestamp=1300387322414, value=REGION => {NAME 
=> 
'domains,1932334:2011/02/18/03:com.photobucket.i654,1299792156289.3824e8b8310176b
  299792156289.3824e8b8310176b6f3c2a1d3f3e708dc. 
6f3c2a1d3f3e708dc.', STARTKEY => 
'1932334:2011/02/18/03:com.photobucket.i654', ENDKEY => 
'1933201:2011/03/02/09:org.wikipedia.af', ENCODED => 3824e8b831017
 
6b6f3c2a1d3f3e708dc, OFFLINE => true, SPLIT => true, TABLE => {{NAME => 
'domains', FAMILIES => [{NAME => 'handling', BLOOMFILTER => 'NONE', 
REPLICATION_SCO
                                                       PE => '0', 
COMPRESSION => 'LZO', VERSIONS => '1', TTL => '1000000000', BLOCKSIZE => 
'65536', IN_MEMORY => 'false', BLOCKCACHE => 'true'}]}}
  domains,1932334:2011/02/18/03:com.photobucket.i654,1 
column=info:server, timestamp=1300757176451, 
value=s8.sjc.opendns.com:60020 

  299792156289.3824e8b8310176b6f3c2a1d3f3e708dc. 
 

  domains,1932334:2011/02/18/03:com.photobucket.i654,1 
column=info:serverstartcode, timestamp=1300757176451, 
value=1300752817197 

  299792156289.3824e8b8310176b6f3c2a1d3f3e708dc. 
 

  domains,1932334:2011/02/18/03:com.photobucket.i654,1 
column=info:splitA, timestamp=1300387322414, value=REGION => {NAME => 
'domains,1932334:2011/02/18/03:com.photobucket.i654,1300387311068.3fbd783ab2a3de505fd
  299792156289.3824e8b8310176b6f3c2a1d3f3e708dc.       5607748d82ec7.', 
STARTKEY => '1932334:2011/02/18/03:com.photobucket.i654', ENDKEY => 
'1932968:2010/11/10/12:com.twitter', ENCODED => 3fbd783ab2a3de505fd560
                                                       7748d82ec7, TABLE 
=> {{NAME => 'domains', FAMILIES => [{NAME => 'handling', BLOOMFILTER => 
'NONE', REPLICATION_SCOPE => '0', COMPRESSION => 'LZO', VERSIONS
                                                        => '1', TTL => 
'1000000000', BLOCKSIZE => '65536', IN_MEMORY => 'false', BLOCKCACHE => 
'true'}]}}
  domains,1932334:2011/02/18/03:com.photobucket.i654,1 
column=info:splitB, timestamp=1300387322414, value=REGION => {NAME => 
'domains,1932968:2010/11/10/12:com.twitter,1300387311068.6e95a3361da531a57b5883014c04
  299792156289.3824e8b8310176b6f3c2a1d3f3e708dc.       7cdc.', STARTKEY 
=> '1932968:2010/11/10/12:com.twitter', ENDKEY => 
'1933201:2011/03/02/09:org.wikipedia.af', ENCODED => 
6e95a3361da531a57b5883014c047cdc, T
                                                       ABLE => {{NAME => 
'domains', FAMILIES => [{NAME => 'handling', BLOOMFILTER => 'NONE', 
REPLICATION_SCOPE => '0', COMPRESSION => 'LZO', VERSIONS => '1', TTL
                                                       => '1000000000', 
BLOCKSIZE => '65536', IN_MEMORY => 'false', BLOCKCACHE => 'true'}]}}


- Adam

Mime
View raw message