hbase-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Ming Ma (Commented) (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HBASE-4497) If region opening fails after updating META HBCK reports it as inconsistent and scanning the region throws NSRE
Date Thu, 29 Sep 2011 03:34:45 GMT

    [ https://issues.apache.org/jira/browse/HBASE-4497?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13116981#comment-13116981
] 

Ming Ma commented on HBASE-4497:
--------------------------------

Using startcode and timestamp is a good idea. However, I want to confirm if there is a case
where it won't work. Given there is no such thing as global clock, the timestamp value generated
by the RS that hosts .META. region at that moment might not be unique if .META. region is
moved to another RS. So there is a possibility of "startcode and timestamp is what i initially
thought of. seems like there could be some weird situations. for example, what is to say that
the server already in META didn't somehow become the new assignment destination?". Here is
how:

1. For a given region, .META. table has RS1 as RS serverName, T1 as timestamp value. { RS1,
T1 }
2. .META. is moved to another RS whose clock is behind after the original RS that wrote {RS1,
T1}.
3. RS2 starts openRegion first, it has an older ZK node version to check. RS1 start openRegion
later. It has an up-to-date ZK node version.
4. Both RS2 and RS1 are about the do checkAndPut on .META. table.  Both will use {RS1, T1}
as condition for checkAndPut.
5. RS1 updates it first, it succeeds. There is a chance that after the update, the value is
still {RS1, T1}, given T1 is generated by a RS whose clock is behind.
6. RS2 updates it next, it also succeeds, given {RS1, T1} hasn't change even RS1 makes an
update earlier.
7. RS1 has the up-to-date ZK node version, thus it will continue and succeeds with the rest
of open operatioin. The region is considered OPENED from AM's point of view.
8. RS2 has older ZK node version, thus will fail later when it tries to update ZK node. Region
won't be opened on RS2.
9. In .META. table, the region is on RS2.


Adding support for version check in checkAndPut should address such scenario.


Regarding the "region assignment ID" approach:

1. I didn't imply it will only be incremented by the Master. I suggested a ZK-based AtomicLong
that Master and all RSs can get hold off. So this could be considered a global clock.
2. Such ID could also help to track all the region transition events, HBASE-4354.


                
> If region opening fails after updating META HBCK reports it as inconsistent and scanning
the region throws NSRE
> ---------------------------------------------------------------------------------------------------------------
>
>                 Key: HBASE-4497
>                 URL: https://issues.apache.org/jira/browse/HBASE-4497
>             Project: HBase
>          Issue Type: Bug
>            Reporter: ramkrishna.s.vasudevan
>            Priority: Critical
>
> As per the discussion in the mail chain "HBCK reporting of possible mismatch in RS assignment"
this JIRA is created.
> Consider two RS-> RS1 and RS2.
> A region tries to open in RS1. But it takes a while.  The RS1 has still not updated meta
and transitioned the node from OPENING to OPENED
> So timeout assigns the region to RS2.  RS2 successfully updates the META and opens the
region.
> Now RS1 tries to act on the region by first updating the META and then transiting the
node to OPENING to OPENED.
> RS1 transiting the node to OPENING to OPENED will fail.  But the META entry will have
RS1 as the latest.
> Now HBCK reports this as an inconsistency and if we try to scan the Region we get NotServingRegionException.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

Mime
View raw message