hadoop-common-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Bikas Saha (Commented) (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HADOOP-8163) Improve ActiveStandbyElector to provide hooks for fencing old active
Date Fri, 23 Mar 2012 01:36:22 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-8163?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13236265#comment-13236265

Bikas Saha commented on HADOOP-8163:

bq. This scenario is impossible for the following reason: If the state of the world changed
and this node was no longer active, the only possible reason for that is that the node lost
its ZK session lease. If that's the case, then it won't be able to issue any further commands
from that client (see my conversation with Hari above)
May not be impossible. I will try to think of a better scenario. But of the top of my head
I came up with this. Lets say A was active. B and C were standby's. (HDFS may have only A
and B but this is a generic lib). B gets the lock but has delay in fencing A. Downtime is
getting high. So paranoid admin deletes the lock hoping a new master might solve this. C gets
lock and fences A. B did not write a lock. So C thinks it has fenced all masters and becomes
master. Now dofence on B completes (fail/success does not matter). It deletes C's breadcrumb.
Now it writes its own new breadcrumb. We have inconsistent znode state where lock znode and
breadcrumb znode are from different sources. Then B asks its service to become active. We
have 2 masters. Now B processes the ZK event that tells it that it has lost the lock. B asks
service to become standby. And deletes its breadcrumb. Now there is no breadcrumb. Now C goes
into bad state. C loses master lock but does not die. And A takes lock. No breadcrumb and
so A becomes master without fencing bad C. A and C are concurrent masters.

bq.I'm currently handling this situation at the ZKFailoverController level. Are you suggesting
we determine "self" by comparing the actual bytewise data of the znode, and skip the fence
call if it's the same as this elector instance's appData? Seems reasonable, just want to clarify.
Yes. I am suggesting to do this within the Elector and not at the ZKFailoverController level.
The "self" compare approach would be reasonable as long as we can assure ourselves that appData
will not be same across different candidates.
> Improve ActiveStandbyElector to provide hooks for fencing old active
> --------------------------------------------------------------------
>                 Key: HADOOP-8163
>                 URL: https://issues.apache.org/jira/browse/HADOOP-8163
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: ha
>    Affects Versions: 0.23.3, 0.24.0
>            Reporter: Todd Lipcon
>            Assignee: Todd Lipcon
>         Attachments: hadoop-8163.txt, hadoop-8163.txt, hadoop-8163.txt, hadoop-8163.txt
> When a new node becomes active in an HA setup, it may sometimes have to take fencing
actions against the node that was formerly active. This JIRA extends the ActiveStandbyElector
which adds an extra non-ephemeral node into the ZK directory, which acts as a second copy
of the active node's information. Then, if the active loses its ZK session, the next active
to be elected may easily locate the unfenced node to take the appropriate actions.

This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira


View raw message