hbase-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "rajeshbabu (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HBASE-8127) Region of a disabling or disabled table could be stuck in transition state when RS dies during Master initialization
Date Mon, 18 Mar 2013 05:38:16 GMT

    [ https://issues.apache.org/jira/browse/HBASE-8127?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13604879#comment-13604879
] 

rajeshbabu commented on HBASE-8127:
-----------------------------------

[~jeffreyz]
Thanks for review.
bq. . I didn't see you handle disabled case so the test case will fail again if I set disabledTable
to disabled in the test case.
Ideally if a table is disabled then the table regions should not be in transition or online
thats why not calling ZKAssign#deleteNodeFailSilent on each region otherwise SSH processing
will take more time if disabled table regions present on deadserver as for the META. 

bq. Once a while I saw the opening RIT of the disabled table either got assigned or stuck
there so I think it's better to add some safe guards when we deal with RITs during a master
node starts up. 
Ideally it should not happen we need to dig more into this when it is possible. If possible
can you give some details about this?

bq.  I put my feedbacks in the feedback patch for your references 
Thanks for your feedback patch as well. I will check this in more detail. 
                
> Region of a disabling or disabled table could be stuck in transition state when RS dies
during Master initialization
> --------------------------------------------------------------------------------------------------------------------
>
>                 Key: HBASE-8127
>                 URL: https://issues.apache.org/jira/browse/HBASE-8127
>             Project: HBase
>          Issue Type: Bug
>    Affects Versions: 0.94.5
>            Reporter: Jeffrey Zhong
>            Assignee: Jeffrey Zhong
>             Fix For: 0.94.7
>
>         Attachments: HBASE-8127_feedback.patch, HBASE-8127.patch, hbase-8127_v1.patch,
reproduce-hang.patch
>
>
> The issue happens when a RS dies during a master starts up. After the RS reports open
to the new master instance and dies immediately thereafter, the RITs of disabling tables(or
disabled table) on the died RS will be in RIT state forever.
> I attached a patch to simulate the situation and you can run the following command to
reproduce the issue:
> {code}mvn test -PlocalTests -Dtest=TestMasterFailover#testMasterFailoverWithMockedRITOnDeadRS{code}
> Basically, we skip regions of a dead server inside AM.processDeadServersAndRecoverLostRegions
as the following code and relies on SSH to process those skipped regions:
> {code}
>           for (Pair<HRegionInfo, Result> deadRegion : deadServer.getValue()) {
>             nodes.remove(deadRegion.getFirst().getEncodedName());
>           }
> {code} 
> While in SSH, we skip regions of disabling(or disabled table) again by function processDeadRegion.
Finally comes to the issue that RITs of disabling(or disabled table) stuck there forever.
>  

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

Mime
View raw message