Mailing-List: contact user-help@hbase.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@hbase.apache.org
Received-SPF: pass (nike.apache.org: domain of m9suns@gmail.com designates
 74.125.83.176 as permitted sender)
MIME-Version: 1.0
Date: Wed, 6 Jul 2011 00:58:40 +0800
Message-ID: 
 <CACVLn7PZzFrDAAspUng2gsKJny-PcaxFZPvStUwOgH14aPSKzQ@mail.gmail.com>
Subject: WrongRegionException and inconsistent table found
From: Xu-Feng Mao <m9suns@gmail.com>
To: user@hbase.apache.org, hbase-user@hadoop.apache.org
Cc: m9suns@gmail.com
Content-Type: multipart/alternative; boundary=bcaec520ef5f5562fb04a75565ec

--bcaec520ef5f5562fb04a75565ec
Content-Type: text/plain; charset=ISO-8859-1

Hi,

We're running a hbase cluster including 37 regionservers. Today, we found
losts of WrongRegionException when putting object into it.

hbase hbck -details
    reports that
====
Chain of regions in table STable is broken; edges does not contain
ztxrGmCwn-6BE32s3cX1TNeHU_I=
ERROR: Found inconsistency in table STable
====

echo "scan '.META.'"| hbase shell &> meta.txt
grep -A1 "STARTKEY => 'EStore_everbox_z" meta.txt
    reports that
====
 Ck=,1308802977279.71ffb1             1ffb10b8b95fd47b3eff468d00ab4e9.',
STARTKEY => 'ztn0ukLW
 0b8b95fd47b3eff468d00ab4             d1NSU3fuXKkkWq5ZVCk=', ENDKEY =>
'ztqdVD8fCMP-dDbXUAydan
 e9.                                                kboD4=', ENCODED =>
71ffb10b8b95fd47b3eff468d00ab4e9, TABLE => {{NAME =
--
 D4=,1305619724446.c45191           45191821053d03537596f4a2e759718.',
STARTKEY => ztqdVD8f
 821053d03537596f4a2e7597           CMP-dDbXUAydankboD4=', ENDKEY => '
ztxrGmCwn-6BE32s3cX1TN
 18.                                                eHU_I=', ENCODED =>
c45191821053d03537596f4a2e759718, TABLE => {{NAME =
--
 pA=,1309455605341.c5c5f5            5c5f578722ea3f8d1b099313bec8298.',
STARTKEY => 'zu3zVaLc
 78722ea3f8d1b099313bec82           GDnnpjKCbnboXgAFspA=', ENDKEY =>
'zu7qkr5fH6MMJ3GxbCv_0d
 98.                                                6g8yI=', ENCODED =>
c5c5f578722ea3f8d1b099313bec8298, TABLE => {{NAME =
====

It looks like the meta indeed has a hole.(We tried scan '.META.' several
times, to confirm it's not a transient status.)
We've tried hbase hbck -fix, does not help.

We found a thread 'wrong region exception' about two months ago. Stack
suggested a 'little surgery' like
====

*So, make sure you actually have a hole.  Dump out your meta table:

echo "scan '.META.'"| ./bin/hbase shell &> /tmp/meta.txt

Then look ensure that there is a hole between the above regions
(compare start and end keys... the end key of one region needs to
match the start key of the next).

If indeed a hole, you need to do a little surgery inserting a new
missing region (hbck should fix this but it doesn't have the smarts
just yet).

Basically, you create a new region with start and end keys to fill the
hole then you insert it into .META. and then assign it.  There are
some scripts in our bin directory that do various parts of this.  I'm
pretty sure its beyond any but a few figuring this mess out so if you
do the above foot work and provide a few more details, I'll hack up
something for you (and hopefully something generalized to be use by
others later, and later to be integrated into hbck).*

====

Can anyone give a detailed example, step by step instruction would be
greatly appreciated.
My understand is we should
1.Since we already has the lost region, we now have start and end keys.
2.generate the row represents the missing region. But how can I generate the
encoded name?
It looks like I need
column=info:server,column=info:serverstartcode and column=info:regioninfo
for the missing region.
And column=info:regioninfo includes so many information. How to generate
them one by one?
As for the name of row, it consists of tablename, startkey, encode, and one
more long number,
how to get this number?
3.use assing command in the hbase shell

We also tried check_meta.rb --fix, it reports
====
11/07/06 00:09:08 WARN check_meta: hole after REGION => {NAME =>
'STable,ztqdVD8fCMP-dDbXUAydankboD4=,1305619724446.c45191821053d03537596f4a2e759718.',
STARTKEY => 'ztqdVD8fCMP-dDbXUAydankboD4=', ENDKEY =>
'ztxrGmCwn-6BE32s3cX1TNeHU_I=', ENCODED => c45191821053d03537596f4a2e759718,
TABLE => {{NAME => 'STable', FAMILIES => [{NAME => 'file', BLOOMFILTER =>
'NONE', REPLICATION_SCOPE => '0', COMPRESSION => 'NONE', VERSIONS => '3',
TTL => '2147483647', BLOCKSIZE => '65536', IN_MEMORY => 'false', BLOCKCACHE
=> 'true'}, {NAME => 'filelength', BLOOMFILTER => 'NONE', REPLICATION_SCOPE
=> '0', COMPRESSION => 'NONE', VERSIONS => '3', TTL => '2147483647',
BLOCKSIZE => '65536', IN_MEMORY => 'false', BLOCKCACHE => 'true'}, {NAME =>
'userbucket', BLOOMFILTER => 'NONE', REPLICATION_SCOPE => '0', COMPRESSION
=> 'NONE', VERSIONS => '3', TTL => '2147483647', BLOCKSIZE => '65536',
IN_MEMORY => 'false', BLOCKCACHE => 'true'}, {NAME => 'userpass',
BLOOMFILTER => 'NONE', REPLICATION_SCOPE => '0', COMPRESSION => 'NONE',
VERSIONS => '3', TTL => '2147483647', BLOCKSIZE => '65536', IN_MEMORY =>
'false', BLOCKCACHE => 'true'}]}}
11/07/06 00:28:40 WARN check_meta: Missing .regioninfo: hdfs://
hd0013.c.gj.com:9000/hbase/STable/3e6faca40a7ccad7ed8c0b5848c0f945/.regioninfo
====

The problem is still there. BTW, what about the blue warning? Is this a
serious issue?
The situation is quite hard to us, it looks like even we can fill the hole
in the meta, we would lost all the data in the hole region, right?

Thanks and regards,

Mao Xu-Feng

--bcaec520ef5f5562fb04a75565ec--