hadoop-common-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Todd Lipcon (Commented) (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HADOOP-8217) Edge case split-brain race in ZK-based auto-failover
Date Sat, 31 Mar 2012 01:57:31 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-8217?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13242944#comment-13242944
] 

Todd Lipcon commented on HADOOP-8217:
-------------------------------------

bq. I would like to question the value of FC2 calling NN1.transitionToStandby() in general.
FC1 on NN1 is supposed to call NN1.transitionToStandby() because thats is FC1's responsibility
upon losing the leader lock.

This doesn't work, since FC1 can take arbitrarily long to notice that it has lost its lock.

bq. Secondly, based on the recent work done to add breadcrumbs to the ActiveStandbyElector,
FC2 is going to fence NN1 if NN1 has not gracefully given up the lock, which is clearly the
case here. So the problem is already solved unless I am mistaken.

But the first stage of "fencing" is to gracefully ask the NN to go to standby. This is exactly
the problem here. If, instead, we always required that we always use an aggressive fencing
mechanism (STONITH/NAS fencing), you're right that there would not be a problem. But we can
avoid that in many cases -- for example, imagine that the active node loses its connection
to the ZK quorum, but still has a connection to the other NN (eg by a crossover cable). In
this case it will leave its breadcrumb znode there, but the new active can easily transition
it to standby.

Here's another way of looking at this JIRA:
- the "aggressive" fencing mechanisms have the property of being "persistent". i.e after fencing,
the node cannot become active, even if asked to.
- the "graceful" fencing mechanism (transitionToStandby() RPC) does not currently have the
property of being "persistent". If another older node asks it to become active after it's
been "gracefully fenced", it will do so incorrectly.
- This JIRA makes "graceful fencing" persistent, so it can be used correctly.


Regarding the ActiveStandbyElector callback for {{becomeStandby}}, I actually think it's redundant.
There are two cases in which it could be called:
- If already standby, it's a no-op
- If active, then this indicates that the elector lost its znode. Since it lost its znode
(rather than quitting the election gracefully), it will leave its breadcrumb behind. Thus,
the other node will fence it. So, calling transitionToStandby is redundant with fencing which
the other node will have to perform anyway.
                
> Edge case split-brain race in ZK-based auto-failover
> ----------------------------------------------------
>
>                 Key: HADOOP-8217
>                 URL: https://issues.apache.org/jira/browse/HADOOP-8217
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: auto-failover, ha
>    Affects Versions: 0.24.0
>            Reporter: Todd Lipcon
>            Assignee: Todd Lipcon
>         Attachments: hadoop-8217-testcase.txt
>
>
> As discussed in HADOOP-8206, the current design for automatic failover has the following
race:
> - ZKFC1 gets active lock
> - ZKFC1 is about to send transitionToActive() and machine freezes (eg GC pause + swapping)
> - ZKFC1 loses its ZK lock, ZKFC2 gets ZK lock
> - ZKFC2 calls transitionToStandby on NN1, and transitions NN2 to active
> - ZKFC1 wakes up from pause, calls transitionToActive(), now we have a bad situation
> This is rare, since it requires ZKFC1 to freeze longer than its ZK session timeout, but
worth fixing, since the results can be disastrous.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

Mime
View raw message