hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jay Talreja <jay.talr...@oracle.com>
Subject Re: Inconsistent Table HBCK
Date Wed, 22 May 2013 10:52:44 GMT

The issue (I think) was related to a region split that didn't happen 
cleanly. As a result there were references to daughter region present in 
I believe the -fixSplitParents would have taken care of this but it is 
not available in 0.94.0

I manually deleted reference files from HDFS and was able to bring the 
table back to consistent state.


On 5/21/13 11:12 PM, Jean-Marc Spaggiari wrote:
> Hi Jay,
> Which 0.94 version are you running? 0.94.0? Or 0.94.7?
> JM
> 2013/5/21 Jay Talreja<jay.talreja@oracle.com>
>> One of our production clusters had several region server failures. As a
>> result one of the tables is in an inconsistent state as reported by hbck.
>> We have tried using hbck repair commands but none seem to work. There is
>> one region that is stuck in a forever pending open state.
>> The error reported in RS log is about a StoreFile not found. But what is
>> really strange is that the store file that is reported as missing does not
>> even belong to the region being opened.
>> We tried to manually create a directory in HDFS and copy the missing file
>> but it causes hbck to report about a region in HDFS but not in Meta.
>> There 4 inconsistencies currently.
>> ERROR: Region { meta =>  <tableName>,I.1521_D.**1361689200_9,1369099149747.
>> **2123fc70fac804cd8d48ea4494cc81**84., hdfs =>  hdfs://host:8020/hbase/**
>> tableName/**2123fc70fac804cd8d48ea4494cc81**84, deployed =>   } not
>> deployed on any region server.
>> ERROR: Region { meta =>  null, hdfs =>  hdfs://hostname:8020/hbase/**
>> tableName/**450ed30b410e9d6d54ac53099039cb**28, deployed =>   } on HDFS,
>> but not listed in META or deployed on any region server
>> 13/05/21 10:51:11 DEBUG util.HBaseFsck: There are 1769 region info entries
>> ERROR: There is a hole in the region chain between I.1521_D.1361689200_9
>> and I.1521_D.1362150000_8.  You need to create a new .regioninfo and region
>> dir in hdfs to plug the hole.
>> ERROR: There is a hole in the region chain between I.1_D.1368392400_9 and
>> I.2020_D.1338948000_2.  You need to create a new .regioninfo and region dir
>> in hdfs to plug the hole.
>> ERROR: Found inconsistency in table<tableName>
>> We are running Hbase 0.94 (Apache) on Hadoop 1.0.3
>> At this stage, we are stuck and are looking for help ! The cluster is in
>> an unbalanced state and region servers frequently keep dying.
>> Thanks,
>> Jay

  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message