hbase-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Craig Condit (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (HBASE-11396) Invalid meta entries can lead to unstartable master
Date Sat, 21 Jun 2014 18:43:24 GMT

     [ https://issues.apache.org/jira/browse/HBASE-11396?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel

Craig Condit updated HBASE-11396:

    Attachment: HBASE-11396-v1.patch

Patch which catches the exception, logs an error, and returns null so that Master can continue

> Invalid meta entries can lead to unstartable master
> ---------------------------------------------------
>                 Key: HBASE-11396
>                 URL: https://issues.apache.org/jira/browse/HBASE-11396
>             Project: HBase
>          Issue Type: Bug
>          Components: master
>    Affects Versions: 0.98.3
>            Reporter: Craig Condit
>         Attachments: HBASE-11396-v1.patch
> Recently I accidentally kill -9'd all regionservers in my cluster (don't ask.. not my
finest moment).
> This let to some corruption of the meta table, causing the following exception to be
output on the HBase Master during startup, and culminating with the Master aborting:
> {noformat}
> java.lang.IllegalArgumentException: Wrong length: 13, expected 8
> 	at org.apache.hadoop.hbase.util.Bytes.explainWrongLengthOrOffset(Bytes.java:600)
> 	at org.apache.hadoop.hbase.util.Bytes.toLong(Bytes.java:578)
> 	at org.apache.hadoop.hbase.HRegionInfo.getServerName(HRegionInfo.java:1059)
> 	at org.apache.hadoop.hbase.HRegionInfo.getHRegionInfoAndServerName(HRegionInfo.java:987)
> 	at org.apache.hadoop.hbase.master.AssignmentManager.rebuildUserRegions(AssignmentManager.java:2678)
> 	at org.apache.hadoop.hbase.master.AssignmentManager.joinCluster(AssignmentManager.java:465)
> 	at org.apache.hadoop.hbase.master.HMaster.finishInitialization(HMaster.java:910)
> 	at org.apache.hadoop.hbase.master.HMaster.run(HMaster.java:615)
> 	at java.lang.Thread.run(Thread.java:744)
> {noformat}
> No tools that I am aware of were able to clean this up. I added a short patch to catch
and log this exception and return null from HRegionInfo.getServerName(). This allowed for
startup of the cluster, and hbase hbck to repair the damage.
> Creating ticket to submit patch in case anyone else finds this useful.

This message was sent by Atlassian JIRA

View raw message