Mailing-List: contact issues-help@hbase.apache.org; run by ezmlm
Precedence: bulk
Date: Sat, 23 Mar 2013 09:27:15 +0000 (UTC)
From: "rajeshbabu (JIRA)" <jira@apache.org>
To: issues@hbase.apache.org
Message-ID: <JIRA.12637408.1363476074596.42242.1364030835908@arcas>
In-Reply-To: <JIRA.12637408.1363476074596@arcas>
References: <JIRA.12637408.1363476074596@arcas>
Subject: [jira] [Commented] (HBASE-8127) Region of a disabling or disabled
 table could be stuck in transition state when RS dies during Master
 initialization
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit


    [ https://issues.apache.org/jira/browse/HBASE-8127?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13611666#comment-13611666 ] 

rajeshbabu commented on HBASE-8127:
-----------------------------------

@Ram,
bq.  I think any exception on it we need not abort the master. Just log and leave it.. it will not cause any harm.
 
If any unexpected zk exceptions we are aborting server every where. Followed the same thing in this patch as well.

bq. Going thro the code again previously before the patch the call to assign would have taken care of this scenario anyway (In AM.offlineDisabledRegion())? The only thing is we are trying to do it once again SSH. Am i right?
In normal case like region server went down after master initialization then we will call assign once again and which will be taken care of deleting znode in AM.offlineDisabledRegion(). But if a RS went down during master initialization no one are deleting/processing znode of disabling table regions because master dont have in-memory state of RIT info, assigned regions info with him. Even assign also getting skipped. 

bq. Also what is the major difference between this JIRA and HBASE-8133.
HBASE-8127 is not a problem in trunk. But unncessary assigns and unassigns happening in SSH if disabling table regions in OPENING/PENDING_OPEN state(HBASE-8133).
HBASE-8133 also possible in 0.94 so combined with HBASE-8127 handled in the latest patch.

                
> Region of a disabling or disabled table could be stuck in transition state when RS dies during Master initialization
> --------------------------------------------------------------------------------------------------------------------
>
>                 Key: HBASE-8127
>                 URL: https://issues.apache.org/jira/browse/HBASE-8127
>             Project: HBase
>          Issue Type: Bug
>    Affects Versions: 0.94.5
>            Reporter: Jeffrey Zhong
>            Assignee: rajeshbabu
>             Fix For: 0.94.7
>
>         Attachments: HBASE-8127_94_2.patch, HBASE-8127_feedback.patch, HBASE-8127.patch, hbase-8127_v1.patch, reproduce-hang.patch
>
>
> The issue happens when a RS dies during a master starts up. After the RS reports open to the new master instance and dies immediately thereafter, the RITs of disabling tables(or disabled table) on the died RS will be in RIT state forever.
> I attached a patch to simulate the situation and you can run the following command to reproduce the issue:
> {code}mvn test -PlocalTests -Dtest=TestMasterFailover#testMasterFailoverWithMockedRITOnDeadRS{code}
> Basically, we skip regions of a dead server inside AM.processDeadServersAndRecoverLostRegions as the following code and relies on SSH to process those skipped regions:
> {code}
>           for (Pair<HRegionInfo, Result> deadRegion : deadServer.getValue()) {
>             nodes.remove(deadRegion.getFirst().getEncodedName());
>           }
> {code} 
> While in SSH, we skip regions of disabling(or disabled table) again by function processDeadRegion. Finally comes to the issue that RITs of disabling(or disabled table) stuck there forever.
>  

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira