Return-Path: X-Original-To: apmail-ambari-dev-archive@www.apache.org Delivered-To: apmail-ambari-dev-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 62C6618256 for ; Wed, 5 Aug 2015 22:36:06 +0000 (UTC) Received: (qmail 99931 invoked by uid 500); 5 Aug 2015 22:36:05 -0000 Delivered-To: apmail-ambari-dev-archive@ambari.apache.org Received: (qmail 99643 invoked by uid 500); 5 Aug 2015 22:36:05 -0000 Mailing-List: contact dev-help@ambari.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@ambari.apache.org Delivered-To: mailing list dev@ambari.apache.org Received: (qmail 99629 invoked by uid 99); 5 Aug 2015 22:36:05 -0000 Received: from arcas.apache.org (HELO arcas.apache.org) (140.211.11.28) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 05 Aug 2015 22:36:05 +0000 Date: Wed, 5 Aug 2015 22:36:05 +0000 (UTC) From: "Alejandro Fernandez (JIRA)" To: dev@ambari.apache.org Message-ID: In-Reply-To: References: Subject: [jira] [Updated] (AMBARI-11743) NameNode is forced to leave safemode, which causes HBMaster master to crash if done too quickly MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394 [ https://issues.apache.org/jira/browse/AMBARI-11743?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alejandro Fernandez updated AMBARI-11743: ----------------------------------------- Fix Version/s: 2.0.3 > NameNode is forced to leave safemode, which causes HBMaster master to crash if done too quickly > ----------------------------------------------------------------------------------------------- > > Key: AMBARI-11743 > URL: https://issues.apache.org/jira/browse/AMBARI-11743 > Project: Ambari > Issue Type: Bug > Components: ambari-server > Affects Versions: 2.1.0 > Reporter: Alejandro Fernandez > Assignee: Alejandro Fernandez > Fix For: 2.1.0, 2.0.3 > > Attachments: AMBARI-11743.patch > > > 1. Install cluster with Ambari 2.1 and HDP 2.3 > 2. Add services HDFS, YARN, MR, ZK, and HBaste > 3. Perform several Stop All and Start All on HDFS service > 4. Periodically, HBase Master will crash > This was a non-HA cluster. > {code} > 2015-06-02 09:34:24,865 WARN [ip-172-31-33-225:16000.activeMasterManager] hdfs.DFSClient: Could not obtain block: BP-925466282-172.31.33.226-1433234647051:blk_1073741829_1005 file=/apps/hbase/data/hbase.id No live nodes contain current block Block locations: Dead nodes: . Throwing a BlockMissingException > 2015-06-02 09:34:24,866 WARN [ip-172-31-33-225:16000.activeMasterManager] hdfs.DFSClient: DFS Read > org.apache.hadoop.hdfs.BlockMissingException: Could not obtain block: BP-925466282-172.31.33.226-1433234647051:blk_1073741829_1005 file=/apps/hbase/data/hbase.id > at org.apache.hadoop.hdfs.DFSInputStream.chooseDataNode(DFSInputStream.java:945) > at org.apache.hadoop.hdfs.DFSInputStream.blockSeekTo(DFSInputStream.java:604) > at org.apache.hadoop.hdfs.DFSInputStream.readWithStrategy(DFSInputStream.java:844) > at org.apache.hadoop.hdfs.DFSInputStream.read(DFSInputStream.java:896) > at java.io.DataInputStream.readFully(DataInputStream.java:195) > at java.io.DataInputStream.readFully(DataInputStream.java:169) > at org.apache.hadoop.hbase.util.FSUtils.getClusterId(FSUtils.java:816) > at org.apache.hadoop.hbase.master.MasterFileSystem.checkRootDir(MasterFileSystem.java:474) > at org.apache.hadoop.hbase.master.MasterFileSystem.createInitialFileSystemLayout(MasterFileSystem.java:146) > at org.apache.hadoop.hbase.master.MasterFileSystem.(MasterFileSystem.java:126) > at org.apache.hadoop.hbase.master.HMaster.finishActiveMasterInitialization(HMaster.java:649) > at org.apache.hadoop.hbase.master.HMaster.access$500(HMaster.java:182) > at org.apache.hadoop.hbase.master.HMaster$1.run(HMaster.java:1646) > at java.lang.Thread.run(Thread.java:745) > 2015-06-02 09:34:24,870 FATAL [ip-172-31-33-225:16000.activeMasterManager] master.HMaster: Failed to become active master > org.apache.hadoop.hdfs.BlockMissingException: Could not obtain block: BP-925466282-172.31.33.226-1433234647051:blk_1073741829_1005 file=/apps/hbase/data/hbase.id > at org.apache.hadoop.hdfs.DFSInputStream.chooseDataNode(DFSInputStream.java:945) > at org.apache.hadoop.hdfs.DFSInputStream.blockSeekTo(DFSInputStream.java:604) > at org.apache.hadoop.hdfs.DFSInputStream.readWithStrategy(DFSInputStream.java:844) > at org.apache.hadoop.hdfs.DFSInputStream.read(DFSInputStream.java:896) > at java.io.DataInputStream.readFully(DataInputStream.java:195) > at java.io.DataInputStream.readFully(DataInputStream.java:169) > at org.apache.hadoop.hbase.util.FSUtils.getClusterId(FSUtils.java:816) > at org.apache.hadoop.hbase.master.MasterFileSystem.checkRootDir(MasterFileSystem.java:474) > at org.apache.hadoop.hbase.master.MasterFileSystem.createInitialFileSystemLayout(MasterFileSystem.java:146) > at org.apache.hadoop.hbase.master.MasterFileSystem.(MasterFileSystem.java:126) > at org.apache.hadoop.hbase.master.HMaster.finishActiveMasterInitialization(HMaster.java:649) > at org.apache.hadoop.hbase.master.HMaster.access$500(HMaster.java:182) > at org.apache.hadoop.hbase.master.HMaster$1.run(HMaster.java:1646) > at java.lang.Thread.run(Thread.java:745) > {code} -- This message was sent by Atlassian JIRA (v6.3.4#6332)