hbase-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Jonathan Gray (Commented) (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HBASE-4497) If region opening fails after updating META HBCK reports it as inconsistent and scanning the region throws NSRE
Date Wed, 28 Sep 2011 00:35:45 GMT

    [ https://issues.apache.org/jira/browse/HBASE-4497?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13116032#comment-13116032
] 

Jonathan Gray commented on HBASE-4497:
--------------------------------------

I was just discussing this scenario with Dhruba a few days back.  There's definitely a race
condition here and I don't see a trivial fix.

We use HLog IO-fencing to ensure that edits don't slip into an HLog after a server is considered
dead by the Master.  But the Master has no way to prevent this META update from slipping in.

We need to make some modification to how the master can safely timeout an OPENING.  One possibility
is for the master to require either an acknowledgment from the RS before moving the region
elsewhere or for the RS to die.  It seems unlikely that we will actually see the RS to Master
acknowledgment since OPENING taking too long is usually a sign of brokenness or the RS being
backed up, I think.  But in any case I'd imagine some kind of OPEN_CANCEL_REQUESTED state
that the Master transitions the node to and only when the RS transitions to OPEN_CANCELED
or OFFLINE or something, then it's safe to reassign elsewhere.

I think this design still has a hole in it though because there are scenarios where the RS
doesn't actually die but for some reason doesn't OPEN or ack the cancel. 

Another option would be to do the RS performed META edits using a CheckAndPut rather than
straight Put.  Or we could move META editing back to the Master where it's easy to do things
atomically :)

The CheckAndPut idea is kind of neat but we'd probably have to send more data on the OPEN_RPC.
 For example, the existing server start code or server name + start code or something guaranteed
unique (guaranteed that a conflicting RS opening stuff wouldn't be able to use the same thing).
 Then the atomicity is on the META region.
                
> If region opening fails after updating META HBCK reports it as inconsistent and scanning
the region throws NSRE
> ---------------------------------------------------------------------------------------------------------------
>
>                 Key: HBASE-4497
>                 URL: https://issues.apache.org/jira/browse/HBASE-4497
>             Project: HBase
>          Issue Type: Bug
>            Reporter: ramkrishna.s.vasudevan
>            Priority: Critical
>
> As per the discussion in the mail chain "HBCK reporting of possible mismatch in RS assignment"
this JIRA is created.
> Consider two RS-> RS1 and RS2.
> A region tries to open in RS1. But it takes a while.  The RS1 has still not updated meta
and transitioned the node from OPENING to OPENED
> So timeout assigns the region to RS2.  RS2 successfully updates the META and opens the
region.
> Now RS1 tries to act on the region by first updating the META and then transiting the
node to OPENING to OPENED.
> RS1 transiting the node to OPENING to OPENED will fail.  But the META entry will have
RS1 as the latest.
> Now HBCK reports this as an inconsistency and if we try to scan the Region we get NotServingRegionException.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

Mime
View raw message