hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From maxjar10 <jcuz...@gmail.com>
Subject WrongRegionException -- add_table.rb screwed up my hbase.
Date Wed, 12 May 2010 01:04:25 GMT

Ok, here's my story in case anyone else encounters the same issue...

My question is this... Why does the table descriptor/meta table information
not match the .regioninfo in each region sub dir? Is this a bad thing? Read
below...

HBase 0.23-1 
Hadoop 0.20.1

So I wanted to add compression to my HBase tables that I already had setup.
So, I went to the hbase shell and ran a alter table set compression to GZ
and decreased the versions from 3 to 2.

I then ran a major_compact on my table to put the change into effect. Even
though this appears to happen instantly I know you need to wait for
fragmentation to drop to 0%.

Now, I wanted to run a job and saw an exception:

org.apache.hadoop.hbase.client.RetriesExhaustedException: Trying to contact
region server Some server, retryOnlyOne=true, index=0, islastrow=true,
tries=9, numtries=10, i=0, listsize=1,
region=Businesses,300120000005:14317197,1269286328987 for region
Businesses,300120000005:14317197,1269286328987, row '300120000013:692', but
failed after 10 attempts.

How very strange... Well, this was odd so I went to HBase shell and ran a
count "Businesses" and it hung (looped, whatever) when it got ~ to the above
start row. So, I cancelled the count and saw a brief message in the datanode
log about the fact that there was a WrongRegionException. Hmmm...

When I looked at the .META. tables I saw that the endRow didn't match up to
the next startRow like it should so it looked as though a region was
missing. Double hmmmm...

Because I clearly had something wrong I decided to try to run the
add_table.rb as suggested in another thread about seeing
WrongRegionExceptions. So, I proceeded to run the add_table.rb only to have
it fail. I modified the script and noticed that it was outputting the
.regioninfo information that is in each regions sub dir. The problem is that
this .regioninfo DOES NOT match the actual table description. There was no
COMPRESSION and the version was back at 3. This was confusing because I
could see in the actual data when I would open the data files that it was
clearly gzipped.

Sigh, so, I went ahead and modified the script to output what does match the
table description and the only thing it uses the .regioninfo files for is
the startKey and the endKey. The rest is the data that actually matches my
alter.

As of right now the count is proceeding passed the row I had a problem with
BUT I'm not sure if the data is actually good. I'll need to scan a few rows.

My question is this... Why does the table descriptor/meta table information
not match the .regioninfo in each region sub dir? Is this a bad thing?

Thanks!
-- 
View this message in context: http://old.nabble.com/WrongRegionException----add_table.rb-screwed-up-my-hbase.-tp28531026p28531026.html
Sent from the HBase User mailing list archive at Nabble.com.


Mime
View raw message