hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From maxjar10 <jcuz...@gmail.com>
Subject Re: WrongRegionException -- add_table.rb screwed up my hbase.
Date Wed, 12 May 2010 06:19:00 GMT

Thanks for the reply St.Ack. No worries, it's the nature of early development
and I look forward to a lot of great things for HBase and Hadoop. Just
wanted to document the issues for posterity purposes.


stack-3 wrote:
> 
> Sorry for trouble caused.  I thought that 0.20.4 added updating of
> .regioninfo on renable of table but I don't see it.  Nonetheless, I'd
> suggest you update to 0.20.4.  It should have fixes at least to save
> your from WRE going forward.
> 
> Thanks for writing the list,
> St.Ack
> 
> On Tue, May 11, 2010 at 9:20 PM, maxjar10 <jcuzens@gmail.com> wrote:
>>
>> Answered my own question. The .regioninfo files are there specifically
>> for
>> performing fsck functionalities like using add_table.rb.
>>
>> The problem is that the .regioninfo files are NOT updated after an alter.
>> This issue is described in:
>>
>> https://issues.apache.org/jira/browse/HBASE-2366
>>
>> The purpose of the .regioninfo files is described here:
>>
>> http://www.larsgeorge.com/2009/10/hbase-architecture-101-storage.html
>>
>>
>> maxjar10 wrote:
>>>
>>> Ok, here's my story in case anyone else encounters the same issue...
>>>
>>> My question is this... Why does the table descriptor/meta table
>>> information not match the .regioninfo in each region sub dir? Is this a
>>> bad thing? Read below...
>>>
>>> HBase 0.23-1
>>> Hadoop 0.20.1
>>>
>>> So I wanted to add compression to my HBase tables that I already had
>>> setup. So, I went to the hbase shell and ran a alter table set
>>> compression
>>> to GZ and decreased the versions from 3 to 2.
>>>
>>> I then ran a major_compact on my table to put the change into effect.
>>> Even
>>> though this appears to happen instantly I know you need to wait for
>>> fragmentation to drop to 0%.
>>>
>>> Now, I wanted to run a job and saw an exception:
>>>
>>> org.apache.hadoop.hbase.client.RetriesExhaustedException: Trying to
>>> contact region server Some server, retryOnlyOne=true, index=0,
>>> islastrow=true, tries=9, numtries=10, i=0, listsize=1,
>>> region=Businesses,300120000005:14317197,1269286328987 for region
>>> Businesses,300120000005:14317197,1269286328987, row '300120000013:692',
>>> but failed after 10 attempts.
>>>
>>> How very strange... Well, this was odd so I went to HBase shell and ran
>>> a
>>> count "Businesses" and it hung (looped, whatever) when it got ~ to the
>>> above start row. So, I cancelled the count and saw a brief message in
>>> the
>>> datanode log about the fact that there was a WrongRegionException.
>>> Hmmm...
>>>
>>> When I looked at the .META. tables I saw that the endRow didn't match up
>>> to the next startRow like it should so it looked as though a region was
>>> missing. Double hmmmm...
>>>
>>> Because I clearly had something wrong I decided to try to run the
>>> add_table.rb as suggested in another thread about seeing
>>> WrongRegionExceptions. So, I proceeded to run the add_table.rb only to
>>> have it fail. I modified the script and noticed that it was outputting
>>> the
>>> .regioninfo information that is in each regions sub dir. The problem is
>>> that this .regioninfo DOES NOT match the actual table description. There
>>> was no COMPRESSION and the version was back at 3. This was confusing
>>> because I could see in the actual data when I would open the data files
>>> that it was clearly gzipped.
>>>
>>> Sigh, so, I went ahead and modified the script to output what does match
>>> the table description and the only thing it uses the .regioninfo files
>>> for
>>> is the startKey and the endKey. The rest is the data that actually
>>> matches
>>> my alter.
>>>
>>> As of right now the count is proceeding passed the row I had a problem
>>> with BUT I'm not sure if the data is actually good. I'll need to scan a
>>> few rows.
>>>
>>> My question is this... Why does the table descriptor/meta table
>>> information not match the .regioninfo in each region sub dir? Is this a
>>> bad thing?
>>>
>>> Thanks!
>>>
>>
>> --
>> View this message in context:
>> http://old.nabble.com/WrongRegionException----add_table.rb-screwed-up-my-hbase.-tp28531026p28531887.html
>> Sent from the HBase User mailing list archive at Nabble.com.
>>
>>
> 
> 

-- 
View this message in context: http://old.nabble.com/WrongRegionException----add_table.rb-screwed-up-my-hbase.-tp28531026p28532368.html
Sent from the HBase User mailing list archive at Nabble.com.


Mime
View raw message