hadoop-common-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Vinayakumar B (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HADOOP-10251) Both NameNodes could be in STANDBY State if SNN network is unstable
Date Fri, 12 Jun 2015 04:31:02 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-10251?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14582914#comment-14582914
] 

Vinayakumar B commented on HADOOP-10251:
----------------------------------------

bq. 2015-06-12 02:26:02,608 ERROR org.apache.hadoop.ha.ZKFailoverController: Local service
NameNode at zdh196/10.43.156.196:9000 has changed the serviceState to active. Expected was
standby. Quitting election marking fencing necessary.
bq. 2015-06-12 02:27:56,878 ERROR org.apache.hadoop.ha.ZKFailoverController: Local service
NameNode at zdh195/10.43.156.195:9000 has changed the serviceState to active. Expected was
standby. Quitting election marking fencing necessary.

The use case mentioned is normal auto failover. Not the manual failover using haadmin commands.
And I am seeing both NN1 and NN2 are not staying in Active mode if the transition happens
from standby->active. This is strange.

Can you check this ?

1. Stop both ZKFCs. and NNs.
2. Start only one ZKFC and NN. It should successfully convert to active and check whether
it is staying in ACTIVE for long time.
3. Attach the logs for ZKFC after restart, (logs from the restarted point, all lines)
4. Also attach the ZKFailoverController.java source code you are using.

> Both NameNodes could be in STANDBY State if SNN network is unstable
> -------------------------------------------------------------------
>
>                 Key: HADOOP-10251
>                 URL: https://issues.apache.org/jira/browse/HADOOP-10251
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: ha
>    Affects Versions: 2.2.0
>            Reporter: Vinayakumar B
>            Assignee: Vinayakumar B
>            Priority: Critical
>             Fix For: 2.5.0
>
>         Attachments: HADOOP-10251.patch, HADOOP-10251.patch, HADOOP-10251.patch, HADOOP-10251.patch,
HADOOP-10251.patch
>
>
> Following corner scenario happened in one of our cluster.
> 1. NN1 was Active and NN2 was Standby
> 2. NN2 machine's network was slow 
> 3. NN1 got shutdown.
> 4. NN2 ZKFC got the notification and trying to check for old active for fencing. (This
took little more time, again due to slow network)
> 5. In between, NN1 got restarted by our automatic monitoring, and ZKFC made it Active.
> 6. Now NN2 ZKFC got Old Active as NN1 and it did graceful fencing of NN1 to STANBY.
> 7. Before writing ActiveBreadCrumb to ZK, NN2 ZKFC got session timeout and got shutdown
before making NN2 Active.
> *Now cluster having both NameNodes as STANDBY.*
> NN1 ZKFC still thinks that its nameNode is in Active state. 
> NN2 ZKFC waiting for election.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message