hadoop-common-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Todd Lipcon (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HADOOP-9608) ZKFC should abort if it sees an unrecognized NN become active
Date Thu, 30 May 2013 22:57:23 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-9608?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13670925#comment-13670925

Todd Lipcon commented on HADOOP-9608:

Right, the full sequence of events as I understand them was:

- NNA is active, NNB is standby.
- NNB's hardware is getting retired, so it is removed, and a new node NNC brought up.
- Configs updated with NNC's address, but NNA's ZKFC is not modified
- Manual failover occurred to NNC
- Some time later, NNC fails, and the automatic failover does not succeed because NNA's ZKFC
still has NNB's address.

> ZKFC should abort if it sees an unrecognized NN become active
> -------------------------------------------------------------
>                 Key: HADOOP-9608
>                 URL: https://issues.apache.org/jira/browse/HADOOP-9608
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: ha
>    Affects Versions: 3.0.0
>            Reporter: Todd Lipcon
> We recently had an issue where one NameNode and ZKFC was updated to a new configuration/IP
address but the ZKFC on the other node was not rebooted. Then, next time a failover occurred,
the second ZKFC was not able to become active because the data in the ActiveBreadCrumb didn't
match the data in its own configuration:
> {code}
> org.apache.hadoop.ha.ActiveStandbyElector: Exception handling the winning of election
> java.lang.IllegalArgumentException: Unable to determine service address for namenode
> {code}
> To prevent this from happening, whenever the ZKFC sees a new NN become active, it should
check that it's properly able to instantiate a ServiceTarget for it, and if not, abort (since
this ZKFC wouldn't be able to handle a failover successfully)

This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

View raw message