hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Zheng Shen <zhengshe...@outlook.com>
Subject Could not initialize all stores for the region
Date Thu, 31 Mar 2016 14:58:51 GMT

Our Hbase cannot performance any write operation while the read operation are fine. I found
the following error from regision server log

Could not initialize all stores for the region=vocabulary,576206_6513944,1459420417369.19faeb6e4da0b1873f68da271b0f5788.

Failed open of region=vocabulary,576206_6513944,1459420417369.19faeb6e4da0b1873f68da271b0f5788.,
starting to roll back the global memstore size.
java.io.IOException: java.io.IOException: java.io.FileNotFoundException: File does not exist:
        at org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(INodeFile.java:66)
        at org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(INodeFile.java:56)
        at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocationsUpdateTimes(FSNamesystem.java:1932)
        at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocationsInt(FSNamesystem.java:1873)
        at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:1853)
        at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:1825)
        at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.getBlockLocations(NameNodeRpcServer.java:559)

Opening of region {ENCODED => 19faeb6e4da0b1873f68da271b0f5788, NAME => 'vocabulary,576206_6513944,1459420417369.19faeb6e4da0b1873f68da271b0f5788.',
STARTKEY => '576206_6513944', ENDKEY => '599122_6739914'} failed, transitioning from
OPENING to FAILED_OPEN in ZK, expecting version 22

We are using Cloudera CDH 5.4.7, the HBase version is 1.0.0-cdh_5.4.7, with HDFS HA enabled
(one of the namenode is running on the server being shutdown). Our HBase cluster expereienced
an expected node shutdown today for about 4 hours. The node which is shutdown hosts the HDFS
namenode and datanode, Cloudera Manager, as well as HBase master and region server (5 nodes
in totally in our small clusder).  During the node shuting down, beside the services running
that that node, the other HDFS namenode, failover server, and 2 of 3 journal node are also
down. After the node is recovered, we restarted the whole CDH cluster, and then it ends like
this one...

The HDFS checking "hdfs fsck" does not report any corrupted blocks.

Any suggesion about where we should look into for this problem?



  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message