hbase-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "stack (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HBASE-21192) Add HOW-TO repair damaged AMv2.
Date Tue, 02 Oct 2018 05:19:00 GMT

    [ https://issues.apache.org/jira/browse/HBASE-21192?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16635045#comment-16635045
] 

stack commented on HBASE-21192:
-------------------------------

h2. Master startup cannot progress, in holding-pattern until region onlined

If the cluster comes up and reports in the logs lines like the below:

{code}
2018-10-01 22:07:42,792 WARN org.apache.hadoop.hbase.master.HMaster: hbase:meta,,1.1588230740
is NOT online; state={1588230740 state=CLOSING, ts=1538456302300, server=ve1017.halxg.cloudera.com,22101,1538449648131};
ServerCrashProcedures=true. Master startup cannot progress, in holding-pattern until region
onlined.
{code}

... there is no procedure to assign meta.

To inject one, use the hbck2 tool:

{code}
 HBASE_CLASSPATH_PREFIX=./hbase-hbck2-1.0.0-SNAPSHOT.jar hbase org.apache.hbase.HBCK2 unassigns
1588230740
{code}

(1588230740 is the hard-coded encoded region name for hbase:meta -- the hbck2 takes encoded
region names).

You'll probably have to assign the hbase:namespace too if you had to assign meta. Look out
for the encoded name of the namespace region... it'll be a line like this:

{code}
2018-10-01 22:09:49,681 WARN org.apache.hadoop.hbase.master.HMaster: hbase:namespace,,1526694055629.37cc206fe9c4bc1c0a46a34c5f523d16.
is NOT online; state={37cc206fe9c4bc1c0a46a34c5f523d16 state=OPEN, ts=1538456987236, server=ve1233.halxg.cloudera.com,22101,1538441741767};
ServerCrashProcedures=true. Master startup cannot progress, in holding-pattern until region
onlined.
{code}

37cc206fe9c4bc1c0a46a34c5f523d16 is the encoded name of the namespace table region... 

(This stuff will be cleaned up more... just dropping note here for moment so don't forget
when doing writeup...)


> Add HOW-TO repair damaged AMv2.
> -------------------------------
>
>                 Key: HBASE-21192
>                 URL: https://issues.apache.org/jira/browse/HBASE-21192
>             Project: HBase
>          Issue Type: Sub-task
>          Components: amv2
>            Reporter: stack
>            Assignee: stack
>            Priority: Major
>
> Need a page or two on how to do various fixups. Will include doc on how to identify particular
circumstance, how to run a repair, as well as caveats (e.g. if no log recovery, then region
may be missing edits).
> Add pointer to log messages, especially those that explicitly ask for operator intervention;
e.g. Master#inMeta.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Mime
View raw message