hadoop-common-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Uma Maheswara Rao G (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HADOOP-10251) Both NameNodes could be in STANDBY State if SNN network is unstable
Date Thu, 17 Apr 2014 15:53:15 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-10251?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13973090#comment-13973090
] 

Uma Maheswara Rao G commented on HADOOP-10251:
----------------------------------------------

I think your idea will work and patch almost looks good to me.

One question:
{code}
 /**
+   * Callback interface for service state change events.
+   * 
+   * This interface is called whenever there is a change in the service state.
+   */
{code}
Seems like this interface will be called on every health monitor status check. But doc says
its a service state changed event. It is exposing impl details as you do that state comparisions
in implementation and do necessary actions on state change. So, at this interface level, we
are not sure whether service state changed from last state or not right. Can you update this
something like Service state notification?

> Both NameNodes could be in STANDBY State if SNN network is unstable
> -------------------------------------------------------------------
>
>                 Key: HADOOP-10251
>                 URL: https://issues.apache.org/jira/browse/HADOOP-10251
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: ha
>    Affects Versions: 2.2.0
>            Reporter: Vinayakumar B
>            Assignee: Vinayakumar B
>            Priority: Critical
>         Attachments: HADOOP-10251.patch, HADOOP-10251.patch, HADOOP-10251.patch, HADOOP-10251.patch
>
>
> Following corner scenario happened in one of our cluster.
> 1. NN1 was Active and NN2 was Standby
> 2. NN2 machine's network was slow 
> 3. NN1 got shutdown.
> 4. NN2 ZKFC got the notification and trying to check for old active for fencing. (This
took little more time, again due to slow network)
> 5. In between, NN1 got restarted by our automatic monitoring, and ZKFC made it Active.
> 6. Now NN2 ZKFC got Old Active as NN1 and it did graceful fencing of NN1 to STANBY.
> 7. Before writing ActiveBreadCrumb to ZK, NN2 ZKFC got session timeout and got shutdown
before making NN2 Active.
> *Now cluster having both NameNodes as STANDBY.*
> NN1 ZKFC still thinks that its nameNode is in Active state. 
> NN2 ZKFC waiting for election.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

Mime
View raw message