Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
Reply-To: common-issues@hadoop.apache.org
Date: Fri, 30 Mar 2012 22:37:28 +0000 (UTC)
From: "Hari Mankude (Commented) (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: 
 <1463769845.40415.1333147048837.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: 
 <921246741.19972.1332800666695.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-8217) Edge case split-brain race in
 ZK-based auto-failover
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit


    [ https://issues.apache.org/jira/browse/HADOOP-8217?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13242818#comment-13242818 ] 

Hari Mankude commented on HADOOP-8217:
--------------------------------------

Todd, 

I don't think zxid will fix the problem. Caveat is that I don't know the exact design that is being implemented here.

Consider the scenario

1. ZKFC1 goes to gc sleep and loses the active lock
2. NN1 also goes to gc sleep. (NN1 was already active)
3. ZKFC2 tries to do transitionToStandby() on NN1. RPC times out.
4. Don't know what happens now in your design
5. Assuming ZKFC2 continues to make NN2 active.
6. NN1 wakes up, assumes that it is active.
7. both NN1 and NN2 are active.

Without some sort of persistent fencing across all shared resources, it will not work.


> Edge case split-brain race in ZK-based auto-failover
> ----------------------------------------------------
>
>                 Key: HADOOP-8217
>                 URL: https://issues.apache.org/jira/browse/HADOOP-8217
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: auto-failover, ha
>    Affects Versions: 0.24.0
>            Reporter: Todd Lipcon
>            Assignee: Todd Lipcon
>         Attachments: hadoop-8217-testcase.txt
>
>
> As discussed in HADOOP-8206, the current design for automatic failover has the following race:
> - ZKFC1 gets active lock
> - ZKFC1 is about to send transitionToActive() and machine freezes (eg GC pause + swapping)
> - ZKFC1 loses its ZK lock, ZKFC2 gets ZK lock
> - ZKFC2 calls transitionToStandby on NN1, and transitions NN2 to active
> - ZKFC1 wakes up from pause, calls transitionToActive(), now we have a bad situation
> This is rare, since it requires ZKFC1 to freeze longer than its ZK session timeout, but worth fixing, since the results can be disastrous.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira