hbase-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Craig Condit (JIRA)" <j...@apache.org>
Subject [jira] [Created] (HBASE-11396) Invalid meta entries can lead to unstartable master
Date Sat, 21 Jun 2014 18:41:24 GMT
Craig Condit created HBASE-11396:
------------------------------------

             Summary: Invalid meta entries can lead to unstartable master
                 Key: HBASE-11396
                 URL: https://issues.apache.org/jira/browse/HBASE-11396
             Project: HBase
          Issue Type: Bug
          Components: master
    Affects Versions: 0.98.3
            Reporter: Craig Condit


Recently I accidentally kill -9'd all regionservers in my cluster (don't ask.. not my finest
moment).

This let to some corruption of the meta table, causing the following exception to be output
on the HBase Master during startup, and culminating with the Master aborting:

{noformat}
java.lang.IllegalArgumentException: Wrong length: 13, expected 8
	at org.apache.hadoop.hbase.util.Bytes.explainWrongLengthOrOffset(Bytes.java:600)
	at org.apache.hadoop.hbase.util.Bytes.toLong(Bytes.java:578)
	at org.apache.hadoop.hbase.HRegionInfo.getServerName(HRegionInfo.java:1059)
	at org.apache.hadoop.hbase.HRegionInfo.getHRegionInfoAndServerName(HRegionInfo.java:987)
	at org.apache.hadoop.hbase.master.AssignmentManager.rebuildUserRegions(AssignmentManager.java:2678)
	at org.apache.hadoop.hbase.master.AssignmentManager.joinCluster(AssignmentManager.java:465)
	at org.apache.hadoop.hbase.master.HMaster.finishInitialization(HMaster.java:910)
	at org.apache.hadoop.hbase.master.HMaster.run(HMaster.java:615)
	at java.lang.Thread.run(Thread.java:744)
{noformat}

No tools that I am aware of were able to clean this up. I added a short patch to catch and
log this exception and return null from HRegionInfo.getServerName(). This allowed for startup
of the cluster, and hbase hbck to repair the damage.

Creating ticket to submit patch in case anyone else finds this useful.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

Mime
View raw message