Return-Path: X-Original-To: apmail-hadoop-hdfs-issues-archive@minotaur.apache.org Delivered-To: apmail-hadoop-hdfs-issues-archive@minotaur.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 2B374D888 for ; Mon, 4 Feb 2013 04:58:14 +0000 (UTC) Received: (qmail 28115 invoked by uid 500); 4 Feb 2013 04:58:14 -0000 Delivered-To: apmail-hadoop-hdfs-issues-archive@hadoop.apache.org Received: (qmail 28075 invoked by uid 500); 4 Feb 2013 04:58:13 -0000 Mailing-List: contact hdfs-issues-help@hadoop.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: hdfs-issues@hadoop.apache.org Delivered-To: mailing list hdfs-issues@hadoop.apache.org Received: (qmail 28066 invoked by uid 99); 4 Feb 2013 04:58:13 -0000 Received: from arcas.apache.org (HELO arcas.apache.org) (140.211.11.28) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 04 Feb 2013 04:58:13 +0000 Date: Mon, 4 Feb 2013 04:58:13 +0000 (UTC) From: "Vinay (JIRA)" To: hdfs-issues@hadoop.apache.org Message-ID: In-Reply-To: References: Subject: [jira] [Updated] (HDFS-4463) ActiveStandbyElector can join election even before Service HEALTHY, and results in null data at ActiveBreadCrumb MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394 [ https://issues.apache.org/jira/browse/HDFS-4463?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vinay updated HDFS-4463: ------------------------ Description: ActiveStandbyElector can store null at ActiveBreadCrumb in the below race condition. At further all failovers will fail resulting NPE. 1. ZKFC restarted. 2. due to machine busy, first zk connection is expired even before the health monitoring returned the status. 3. On re-establishment transitionToActive will be called, at this time appData will be null, 4. So now ActiveBreadCrumb will have null. 5. After this any failovers will fail throwing {noformat}java.lang.NullPointerException at org.apache.hadoop.util.StringUtils.byteToHexString(StringUtils.java:171) at org.apache.hadoop.ha.ActiveStandbyElector.fenceOldActive(ActiveStandbyElector.java:892) at org.apache.hadoop.ha.ActiveStandbyElector.becomeActive(ActiveStandbyElector.java:797) at org.apache.hadoop.ha.ActiveStandbyElector.processResult(ActiveStandbyElector.java:475) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:545) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:497){noformat} Should not join the election before service is HEALTHY was: ActiveStandbyElector can store null at ActiveBreadCrumb in the below race condition. At further all failovers will fail resulting NPE. 1. ZKFC restarted. 2. due to less machine busy, first zk connection is expired even before the health monitoring returned the status. 3. On re-establishment transitionToActive will be called, at this time appData will be null, 4. So now ActiveBreadCrumb will have null. 5. After this any failovers will fail throwing {noformat}java.lang.NullPointerException at org.apache.hadoop.util.StringUtils.byteToHexString(StringUtils.java:171) at org.apache.hadoop.ha.ActiveStandbyElector.fenceOldActive(ActiveStandbyElector.java:892) at org.apache.hadoop.ha.ActiveStandbyElector.becomeActive(ActiveStandbyElector.java:797) at org.apache.hadoop.ha.ActiveStandbyElector.processResult(ActiveStandbyElector.java:475) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:545) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:497){noformat} Should not join the election before service is HEALTHY > ActiveStandbyElector can join election even before Service HEALTHY, and results in null data at ActiveBreadCrumb > ---------------------------------------------------------------------------------------------------------------- > > Key: HDFS-4463 > URL: https://issues.apache.org/jira/browse/HDFS-4463 > Project: Hadoop HDFS > Issue Type: Bug > Components: ha > Affects Versions: 2.0.2-alpha > Reporter: Vinay > Assignee: Vinay > Priority: Critical > > ActiveStandbyElector can store null at ActiveBreadCrumb in the below race condition. At further all failovers will fail resulting NPE. > 1. ZKFC restarted. > 2. due to machine busy, first zk connection is expired even before the health monitoring returned the status. > 3. On re-establishment transitionToActive will be called, at this time appData will be null, > 4. So now ActiveBreadCrumb will have null. > 5. After this any failovers will fail throwing > {noformat}java.lang.NullPointerException > at org.apache.hadoop.util.StringUtils.byteToHexString(StringUtils.java:171) > at org.apache.hadoop.ha.ActiveStandbyElector.fenceOldActive(ActiveStandbyElector.java:892) > at org.apache.hadoop.ha.ActiveStandbyElector.becomeActive(ActiveStandbyElector.java:797) > at org.apache.hadoop.ha.ActiveStandbyElector.processResult(ActiveStandbyElector.java:475) > at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:545) > at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:497){noformat} > Should not join the election before service is HEALTHY -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira