Mailing-List: contact issues-help@hbase.apache.org; run by ezmlm
Precedence: bulk
Date: Thu, 14 Dec 2017 09:53:05 +0000 (UTC)
From: "Appy (JIRA)" <jira@apache.org>
To: issues@hbase.apache.org
Message-ID: <JIRA.13123621.1512707107000.456665.1513245185857@Atlassian.JIRA>
In-Reply-To: <JIRA.13123621.1512707107000@Atlassian.JIRA>
References: <JIRA.13123621.1512707107000@Atlassian.JIRA> <JIRA.13123621.1512707107748@jira-lw-us.apache.org>
Subject: [jira] [Commented] (HBASE-19457) Debugging flaky
 TestTruncateTableProcedure#testRecoveryAndDoubleExecutionPreserveSplits
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
archived-at: Thu, 14 Dec 2017 09:53:10 -0000


    [ https://issues.apache.org/jira/browse/HBASE-19457?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16290632#comment-16290632 ] 

Appy commented on HBASE-19457:
------------------------------

After more debugging, i think i finally have fix (sorry for being slow, just beginning to understand AM).

So the issue is,
We delete table's state from meta (in step [TRUNCATE_TABLE_REMOVE_FROM_META |https://github.com/apache/hbase/blob/7466e64abb2c68c8a0f40f6051e4b5bf550e69bd/hbase-server/src/main/java/org/apache/hadoop/hbase/master/procedure/TruncateTableProcedure.java#L102])
On recovery, TableStateManager#fixTableStates assumes that missing state means enabled table is enabled. ([here|https://github.com/apache/hbase/blob/7466e64abb2c68c8a0f40f6051e4b5bf550e69bd/hbase-server/src/main/java/org/apache/hadoop/hbase/master/TableStateManager.java#L218])
 
Later we add regions to meta and crash after that. On recovery, AM sees these regions, looks for table state and finds it enabled, and starts assigning them and screws up.

Simple fix here would be: Don't delete table state from meta, just let it remain DISABLED.
-------

But CreateTableProcedure also adds regions to meta and crashes. Why don't we see same issue there?
It adds region row to meta, but does not add any row for the table. 
On recovery, when AM looks for table state corresponding to those regions, TSM#getTableState() throws TableNotFoundException, which get's caught [here|https://github.com/apache/hbase/blob/master/hbase-server/src/main/java/org/apache/hadoop/hbase/master/TableStateManager.java#L135]..etc etc
End result being, it ignores those regions.

----

Some bigger questions to ponder:
1) Should we really assume missing state column as enabled? Probably assuming disabled is more conservative and better choice? Won't screws up the cluster. (Only other place delete the state column is hbck)
2) Shouldn't new regions always be added with state closed? (dev thread: http://mail-archives.apache.org/mod_mbox/hbase-dev/201712.mbox/browser)

> Debugging flaky TestTruncateTableProcedure#testRecoveryAndDoubleExecutionPreserveSplits
> ---------------------------------------------------------------------------------------
>
>                 Key: HBASE-19457
>                 URL: https://issues.apache.org/jira/browse/HBASE-19457
>             Project: HBase
>          Issue Type: Bug
>            Reporter: Appy
>            Assignee: Appy
>         Attachments: patch1, test-output.txt
>
>


--
This message was sent by Atlassian JIRA
(v6.4.14#64029)