ambari-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Alejandro Fernandez (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (AMBARI-11743) NameNode is forced to leave safemode, which causes HBMaster master to crash if done too quickly
Date Sat, 06 Jun 2015 00:23:00 GMT

     [ https://issues.apache.org/jira/browse/AMBARI-11743?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Alejandro Fernandez updated AMBARI-11743:
-----------------------------------------
    Description: 
1. Install cluster with Ambari 2.1 and HDP 2.3
2. Add services HDFS, YARN, MR, ZK, and HBaste
3. Perform several Stop All and Start All on HDFS service
4. Periodically, HBase Master will crash

This was a non-HA cluster.

{code}
2015-06-02 09:34:24,865 WARN  [ip-172-31-33-225:16000.activeMasterManager] hdfs.DFSClient:
Could not obtain block: BP-925466282-172.31.33.226-1433234647051:blk_1073741829_1005 file=/apps/hbase/data/hbase.id
No live nodes contain current block Block locations: Dead nodes: . Throwing a BlockMissingException
2015-06-02 09:34:24,866 WARN  [ip-172-31-33-225:16000.activeMasterManager] hdfs.DFSClient:
DFS Read
org.apache.hadoop.hdfs.BlockMissingException: Could not obtain block: BP-925466282-172.31.33.226-1433234647051:blk_1073741829_1005
file=/apps/hbase/data/hbase.id
	at org.apache.hadoop.hdfs.DFSInputStream.chooseDataNode(DFSInputStream.java:945)
	at org.apache.hadoop.hdfs.DFSInputStream.blockSeekTo(DFSInputStream.java:604)
	at org.apache.hadoop.hdfs.DFSInputStream.readWithStrategy(DFSInputStream.java:844)
	at org.apache.hadoop.hdfs.DFSInputStream.read(DFSInputStream.java:896)
	at java.io.DataInputStream.readFully(DataInputStream.java:195)
	at java.io.DataInputStream.readFully(DataInputStream.java:169)
	at org.apache.hadoop.hbase.util.FSUtils.getClusterId(FSUtils.java:816)
	at org.apache.hadoop.hbase.master.MasterFileSystem.checkRootDir(MasterFileSystem.java:474)
	at org.apache.hadoop.hbase.master.MasterFileSystem.createInitialFileSystemLayout(MasterFileSystem.java:146)
	at org.apache.hadoop.hbase.master.MasterFileSystem.<init>(MasterFileSystem.java:126)
	at org.apache.hadoop.hbase.master.HMaster.finishActiveMasterInitialization(HMaster.java:649)
	at org.apache.hadoop.hbase.master.HMaster.access$500(HMaster.java:182)
	at org.apache.hadoop.hbase.master.HMaster$1.run(HMaster.java:1646)
	at java.lang.Thread.run(Thread.java:745)
2015-06-02 09:34:24,870 FATAL [ip-172-31-33-225:16000.activeMasterManager] master.HMaster:
Failed to become active master
org.apache.hadoop.hdfs.BlockMissingException: Could not obtain block: BP-925466282-172.31.33.226-1433234647051:blk_1073741829_1005
file=/apps/hbase/data/hbase.id
	at org.apache.hadoop.hdfs.DFSInputStream.chooseDataNode(DFSInputStream.java:945)
	at org.apache.hadoop.hdfs.DFSInputStream.blockSeekTo(DFSInputStream.java:604)
	at org.apache.hadoop.hdfs.DFSInputStream.readWithStrategy(DFSInputStream.java:844)
	at org.apache.hadoop.hdfs.DFSInputStream.read(DFSInputStream.java:896)
	at java.io.DataInputStream.readFully(DataInputStream.java:195)
	at java.io.DataInputStream.readFully(DataInputStream.java:169)
	at org.apache.hadoop.hbase.util.FSUtils.getClusterId(FSUtils.java:816)
	at org.apache.hadoop.hbase.master.MasterFileSystem.checkRootDir(MasterFileSystem.java:474)
	at org.apache.hadoop.hbase.master.MasterFileSystem.createInitialFileSystemLayout(MasterFileSystem.java:146)
	at org.apache.hadoop.hbase.master.MasterFileSystem.<init>(MasterFileSystem.java:126)
	at org.apache.hadoop.hbase.master.HMaster.finishActiveMasterInitialization(HMaster.java:649)
	at org.apache.hadoop.hbase.master.HMaster.access$500(HMaster.java:182)
	at org.apache.hadoop.hbase.master.HMaster$1.run(HMaster.java:1646)
	at java.lang.Thread.run(Thread.java:745)
{code}

  was:
1. Install cluster with Ambari 2.1 and HDP 2.3
2. Add services HDFS, YARN, MR, ZK, and HBaste
3. Perform several Stop All and Start All on HDFS service
4. Periodically, HBase Master will crash

This was a non-HA cluster.


> NameNode is forced to leave safemode, which causes HBMaster master to crash if done too
quickly
> -----------------------------------------------------------------------------------------------
>
>                 Key: AMBARI-11743
>                 URL: https://issues.apache.org/jira/browse/AMBARI-11743
>             Project: Ambari
>          Issue Type: Bug
>            Reporter: Alejandro Fernandez
>            Assignee: Alejandro Fernandez
>
> 1. Install cluster with Ambari 2.1 and HDP 2.3
> 2. Add services HDFS, YARN, MR, ZK, and HBaste
> 3. Perform several Stop All and Start All on HDFS service
> 4. Periodically, HBase Master will crash
> This was a non-HA cluster.
> {code}
> 2015-06-02 09:34:24,865 WARN  [ip-172-31-33-225:16000.activeMasterManager] hdfs.DFSClient:
Could not obtain block: BP-925466282-172.31.33.226-1433234647051:blk_1073741829_1005 file=/apps/hbase/data/hbase.id
No live nodes contain current block Block locations: Dead nodes: . Throwing a BlockMissingException
> 2015-06-02 09:34:24,866 WARN  [ip-172-31-33-225:16000.activeMasterManager] hdfs.DFSClient:
DFS Read
> org.apache.hadoop.hdfs.BlockMissingException: Could not obtain block: BP-925466282-172.31.33.226-1433234647051:blk_1073741829_1005
file=/apps/hbase/data/hbase.id
> 	at org.apache.hadoop.hdfs.DFSInputStream.chooseDataNode(DFSInputStream.java:945)
> 	at org.apache.hadoop.hdfs.DFSInputStream.blockSeekTo(DFSInputStream.java:604)
> 	at org.apache.hadoop.hdfs.DFSInputStream.readWithStrategy(DFSInputStream.java:844)
> 	at org.apache.hadoop.hdfs.DFSInputStream.read(DFSInputStream.java:896)
> 	at java.io.DataInputStream.readFully(DataInputStream.java:195)
> 	at java.io.DataInputStream.readFully(DataInputStream.java:169)
> 	at org.apache.hadoop.hbase.util.FSUtils.getClusterId(FSUtils.java:816)
> 	at org.apache.hadoop.hbase.master.MasterFileSystem.checkRootDir(MasterFileSystem.java:474)
> 	at org.apache.hadoop.hbase.master.MasterFileSystem.createInitialFileSystemLayout(MasterFileSystem.java:146)
> 	at org.apache.hadoop.hbase.master.MasterFileSystem.<init>(MasterFileSystem.java:126)
> 	at org.apache.hadoop.hbase.master.HMaster.finishActiveMasterInitialization(HMaster.java:649)
> 	at org.apache.hadoop.hbase.master.HMaster.access$500(HMaster.java:182)
> 	at org.apache.hadoop.hbase.master.HMaster$1.run(HMaster.java:1646)
> 	at java.lang.Thread.run(Thread.java:745)
> 2015-06-02 09:34:24,870 FATAL [ip-172-31-33-225:16000.activeMasterManager] master.HMaster:
Failed to become active master
> org.apache.hadoop.hdfs.BlockMissingException: Could not obtain block: BP-925466282-172.31.33.226-1433234647051:blk_1073741829_1005
file=/apps/hbase/data/hbase.id
> 	at org.apache.hadoop.hdfs.DFSInputStream.chooseDataNode(DFSInputStream.java:945)
> 	at org.apache.hadoop.hdfs.DFSInputStream.blockSeekTo(DFSInputStream.java:604)
> 	at org.apache.hadoop.hdfs.DFSInputStream.readWithStrategy(DFSInputStream.java:844)
> 	at org.apache.hadoop.hdfs.DFSInputStream.read(DFSInputStream.java:896)
> 	at java.io.DataInputStream.readFully(DataInputStream.java:195)
> 	at java.io.DataInputStream.readFully(DataInputStream.java:169)
> 	at org.apache.hadoop.hbase.util.FSUtils.getClusterId(FSUtils.java:816)
> 	at org.apache.hadoop.hbase.master.MasterFileSystem.checkRootDir(MasterFileSystem.java:474)
> 	at org.apache.hadoop.hbase.master.MasterFileSystem.createInitialFileSystemLayout(MasterFileSystem.java:146)
> 	at org.apache.hadoop.hbase.master.MasterFileSystem.<init>(MasterFileSystem.java:126)
> 	at org.apache.hadoop.hbase.master.HMaster.finishActiveMasterInitialization(HMaster.java:649)
> 	at org.apache.hadoop.hbase.master.HMaster.access$500(HMaster.java:182)
> 	at org.apache.hadoop.hbase.master.HMaster$1.run(HMaster.java:1646)
> 	at java.lang.Thread.run(Thread.java:745)
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message