hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Chris Howe <how...@infochimps.org>
Subject Re: Inconsistent META data for a region.
Date Fri, 28 Jan 2011 18:56:06 GMT
Stack <stack@...> writes:

> > ERROR: Region test,,1296067171940.0200bfe58a9e9fadf8ebfa523c47332f. found on
> > server but is listed in META to be on server ip-10-117-
> > 81.ec2.internal:60020.
> Could this region have been deployed on this server before you ran
> add_table?  Or 'test' was a completely new addition.

The region had been deployed, but I dropped the table before I tried to re-add 
> If the latter, then this is a new spin.  add_table.rb effectively just
> edits .META.   Somehow, the assignment of the just-added regions went
> awry.  Can you grep this region name in your master log?  You might be
> able to make some sense of what happened; was the region assigned two
> places?
> > Now, this table is not at all important to me. However, if I drop it, these 
> > regions end up surviving, and hbck gives me a different inconsistency 
> > (... not listed in META...).
> >
> Yeah.  Disable and drop are going by the content in .META.  Thats
> supposed to be the authority.   You could restart these individual
> regionservers (or restart cluster)?  That'd clean up the mess.

When I would stop a single regionserver that was hosting the phantom region, it 
would just migrate to a different one. I managed to get the warnings to 
completely go away for a time this morning, by shutting down all of the 
regionservers. Hbck came back clean.

Unfortunately, I then tried loading a large table that I had created as a set of 
HFiles, and when I ran "loadtable" on them, I was rewarded with 281 regions that 
are now on regionservers and not in META.

> > Is there a way that I can just zot these regions out and make hbase hbck 
> > again? 
> Not in 0.89.x.  I believe you can send a direct close region to the
> individual servers in 0.90 (I'd have to check) so you don't have to do
> full regionserver restart.

I did find the close_region tool in the shell, but I am not sure that it worked 
for me.

> How did you make the table?  Was it with bulk loader?

The "test" table was made with "puts". My new disaster is an HTable.


View raw message