Mailing-List: contact issues-help@hbase.apache.org; run by ezmlm
Precedence: bulk
Date: Thu, 29 Sep 2011 03:34:45 +0000 (UTC)
From: "Ming Ma (Commented) (JIRA)" <jira@apache.org>
To: issues@hbase.apache.org
Message-ID: 
 <1869946466.6222.1317267285502.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: 
 <1263753555.454.1317147225709.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HBASE-4497) If region opening fails after
 updating META HBCK reports it as inconsistent and scanning the region
 throws NSRE
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit


    [ https://issues.apache.org/jira/browse/HBASE-4497?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13116981#comment-13116981 ] 

Ming Ma commented on HBASE-4497:
--------------------------------

Using startcode and timestamp is a good idea. However, I want to confirm if there is a case where it won't work. Given there is no such thing as global clock, the timestamp value generated by the RS that hosts .META. region at that moment might not be unique if .META. region is moved to another RS. So there is a possibility of "startcode and timestamp is what i initially thought of. seems like there could be some weird situations. for example, what is to say that the server already in META didn't somehow become the new assignment destination?". Here is how:

1. For a given region, .META. table has RS1 as RS serverName, T1 as timestamp value. { RS1, T1 }
2. .META. is moved to another RS whose clock is behind after the original RS that wrote {RS1, T1}.
3. RS2 starts openRegion first, it has an older ZK node version to check. RS1 start openRegion later. It has an up-to-date ZK node version.
4. Both RS2 and RS1 are about the do checkAndPut on .META. table.  Both will use {RS1, T1} as condition for checkAndPut.
5. RS1 updates it first, it succeeds. There is a chance that after the update, the value is still {RS1, T1}, given T1 is generated by a RS whose clock is behind.
6. RS2 updates it next, it also succeeds, given {RS1, T1} hasn't change even RS1 makes an update earlier.
7. RS1 has the up-to-date ZK node version, thus it will continue and succeeds with the rest of open operatioin. The region is considered OPENED from AM's point of view.
8. RS2 has older ZK node version, thus will fail later when it tries to update ZK node. Region won't be opened on RS2.
9. In .META. table, the region is on RS2.


Adding support for version check in checkAndPut should address such scenario.


Regarding the "region assignment ID" approach:

1. I didn't imply it will only be incremented by the Master. I suggested a ZK-based AtomicLong that Master and all RSs can get hold off. So this could be considered a global clock.
2. Such ID could also help to track all the region transition events, HBASE-4354.


> If region opening fails after updating META HBCK reports it as inconsistent and scanning the region throws NSRE
> ---------------------------------------------------------------------------------------------------------------
>
>                 Key: HBASE-4497
>                 URL: https://issues.apache.org/jira/browse/HBASE-4497
>             Project: HBase
>          Issue Type: Bug
>            Reporter: ramkrishna.s.vasudevan
>            Priority: Critical
>
> As per the discussion in the mail chain "HBCK reporting of possible mismatch in RS assignment" this JIRA is created.
> Consider two RS-> RS1 and RS2.
> A region tries to open in RS1. But it takes a while.  The RS1 has still not updated meta and transitioned the node from OPENING to OPENED
> So timeout assigns the region to RS2.  RS2 successfully updates the META and opens the region.
> Now RS1 tries to act on the region by first updating the META and then transiting the node to OPENING to OPENED.
> RS1 transiting the node to OPENING to OPENED will fail.  But the META entry will have RS1 as the latest.
> Now HBCK reports this as an inconsistency and if we try to scan the Region we get NotServingRegionException.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira