hbase-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ramkrishna S Vasudevan <ramakrish...@huawei.com>
Subject FW: FW: HBCK reporting of possible mismatch in RS assignment
Date Fri, 23 Sep 2011 14:43:46 GMT

Sorry last msg was mistakenly sent.  HBCK report is also attached here.
Pls find the logs for the problem.  Also when we try scanning we get
exception saying regionnotbeing served.


RS1 
=== 
2011-09-23 22:34:34,000 INFO
org.apache.hadoop.hbase.regionserver.HRegionServer: addToOnlineRegions is
doneREGION => {NAME =>
't5,,1316797380065.2d06b3ca4d398ec96920ae86441a68c9.', TableName => 't5',
STARTKEY => '', ENDKEY => '', ENCODED => 2d06b3ca4d398ec96920ae86441a68c9,} 
2011-09-23 22:34:34,009 INFO org.apache.hadoop.hbase.catalog.MetaEditor:
Updated row t5,,1316797380065.2d06b3ca4d398ec96920ae86441a68c9. in region
.META.,,1 with serverName=linux76,60020,1316796517682 
2011-09-23 22:34:34,009 INFO
org.apache.hadoop.hbase.regionserver.HRegionServer: Done with post open
deploy taks for region=t5,,1316797380065.2d06b3ca4d398ec96920ae86441a68c9.,
daughter=false 
2011-09-23 22:34:34,009 DEBUG org.apache.hadoop.hbase.zookeeper.ZKAssign:
regionserver:60020-0x1328ceaa1ff0037 Attempting to transition node
2d06b3ca4d398ec96920ae86441a68c9 from RS_ZK_REGION_OPENING to
RS_ZK_REGION_OPENED 
2011-09-23 22:34:34,038 WARN
org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler: Completed
the OPEN of region t5,,1316797380065.2d06b3ca4d398ec96920ae86441a68c9. but
when transitioning from  OPENING to OPENED got a version mismatch, someone
else clashed so now unassigning -- closing region 
2011-09-23 22:34:34,038 DEBUG org.apache.hadoop.hbase.regionserver.HRegion:
Closing t5,,1316797380065.2d06b3ca4d398ec96920ae86441a68c9.: disabling
compactions & flushes 
2011-09-23 22:34:34,038 DEBUG org.apache.hadoop.hbase.regionserver.HRegion:
Updates disabled for region
t5,,1316797380065.2d06b3ca4d398ec96920ae86441a68c9. 
2011-09-23 22:34:34,038 DEBUG org.apache.hadoop.hbase.regionserver.Store:
closed f5 
2011-09-23 22:34:34,038 INFO org.apache.hadoop.hbase.regionserver.HRegion:
Closed t5,,1316797380065.2d06b3ca4d398ec96920ae86441a68c9. 

RS2 
=== 
2011-09-23 22:33:56,546 DEBUG org.apache.hadoop.hbase.zookeeper.ZKAssign:
regionserver:60020-0x1328ceaa1ff0039 Successfully transitioned node
2d06b3ca4d398ec96920ae86441a68c9 from RS_ZK_REGION_OPENING to
RS_ZK_REGION_OPENING 
2011-09-23 22:33:56,845 INFO
org.apache.hadoop.hbase.regionserver.HRegionServer: Post open deploy tasks
for region=t5,,1316797380065.2d06b3ca4d398ec96920ae86441a68c9.,
daughter=false 
2011-09-23 22:33:56,845 INFO
org.apache.hadoop.hbase.regionserver.HRegionServer: addToOnlineRegions is
doneREGION => {NAME =>
't5,,1316797380065.2d06b3ca4d398ec96920ae86441a68c9.', TableName => 't5',
STARTKEY => '', ENDKEY => '', ENCODED => 2d06b3ca4d398ec96920ae86441a68c9,} 
2011-09-23 22:33:56,856 INFO org.apache.hadoop.hbase.catalog.MetaEditor:
Updated row t5,,1316797380065.2d06b3ca4d398ec96920ae86441a68c9. in region
.META.,,1 with serverName=linux146,60020,1316796499216 
2011-09-23 22:33:56,856 INFO
org.apache.hadoop.hbase.regionserver.HRegionServer: Done with post open
deploy taks for region=t5,,1316797380065.2d06b3ca4d398ec96920ae86441a68c9.,
daughter=false 
2011-09-23 22:33:58,887 DEBUG org.apache.hadoop.hbase.zookeeper.ZKAssign:
regionserver:60020-0x1328ceaa1ff0039 Attempting to transition node
2d06b3ca4d398ec96920ae86441a68c9 from RS_ZK_REGION_OPENING to
RS_ZK_REGION_OPENED 
2011-09-23 22:33:58,893 DEBUG org.apache.hadoop.hbase.zookeeper.ZKAssign:
regionserver:60020-0x1328ceaa1ff0039 Successfully transitioned node
2d06b3ca4d398ec96920ae86441a68c9 from RS_ZK_REGION_OPENING to
RS_ZK_REGION_OPENED 
2011-09-23 22:33:58,893 DEBUG
org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler: Opened
t5,,1316797380065.2d06b3ca4d398ec96920ae86441a68c9. 


HBCK reports:
============
ERROR: Region t5,,1316797380065.2d06b3ca4d398ec96920ae86441a68c9. listed in
META 
 on region server linux76,60020,1316796517682 but found on region server
linux14 
6,60020,1316796499216


Now try scanning 
================ 
2011-09-23 22:35:32,789 DEBUG
org.apache.hadoop.hbase.regionserver.HRegionServer:
NotServingRegionException; Region is not online:
t5,,1316797380065.2d06b3ca4d398ec96920ae86441a68c9.



-----Original Message-----
From: Todd Lipcon [mailto:todd@cloudera.com] 
Sent: Thursday, September 22, 2011 11:05 AM
To: dev@hbase.apache.org; ramakrishnas@huawei.com
Subject: Re: FW: HBCK reporting of possible mismatch in RS assignment

Hey Ramkrishna,

I agree - sounds like a bug to me.

We need to think more carefully about some of these transitions - I
think we basically have an underspecified incorrect version of
two-phase commit in 5 different places in HBase :)

-Todd

On Thu, Sep 22, 2011 at 1:37 PM, Ramkrishna S Vasudevan
<ramakrishnas@huawei.com> wrote:
> Hi All
>
> Any comments on this.
>
> Best Regards
> Ram
> -----Original Message-----
> From: Ramkrishna S Vasudevan [mailto:ramakrishnas@huawei.com]
> Sent: Wednesday, September 21, 2011 7:46 PM
> To: dev@hbase.apache.org
> Subject: HBCK reporting of possible mismatch in RS assignment
>
> Hi All,
>
>
>
> I have a query regarding the report generated by HBCK.
>
>
>
> Assume there are two RS-  RS1 and RS2.
>
> A region R1 is being opened in RS1.  If it takes more time to open
> particularly in the updateMeta part.  So now if the transition from
OPENING
> to OPENED fails(due to timeout monitor reassigning) though we clear the
info
> of this region in RS1 we do not clear the entry that was updated in Meta.
> So META will have an entry for RS1 hosting R1.
>
>
>
> Now the assignment of R1 to RS2 becomes successful but we have two entries
> in META for the same region.
>
>
>
> If we run the HBCK tool it reports an inconsistency saying region
assignment
> in META is RS1 but actually assigned to RS2.
>
> Is this expected or we need to dig in more and make the HBCK aware of such
> scenarios ?
>
>
>
> Pls correct me if am not correct in my query?
>
>
>
> Best Regards
>
> Ram
>
>
>
>
>
>
>



-- 
Todd Lipcon
Software Engineer, Cloudera


Mime
View raw message