hbase-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Jeffrey Zhong (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (HBASE-8729) distributedLogReplay may hang during chained region server failure
Date Wed, 12 Jun 2013 04:57:21 GMT

     [ https://issues.apache.org/jira/browse/HBASE-8729?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Jeffrey Zhong updated HBASE-8729:
---------------------------------

    Attachment: hbase-8729-v3.patch

Incorporate Stack's first two comments. 
                
> distributedLogReplay may hang during chained region server failure
> ------------------------------------------------------------------
>
>                 Key: HBASE-8729
>                 URL: https://issues.apache.org/jira/browse/HBASE-8729
>             Project: HBase
>          Issue Type: Bug
>          Components: MTTR
>            Reporter: Jeffrey Zhong
>            Assignee: Jeffrey Zhong
>             Fix For: 0.98.0, 0.95.2
>
>         Attachments: 8729-v2.patch, hbase-8729.patch, hbase-8729-v3.patch
>
>
> In a test, half cluster(in terms of region servers) was down and some log replay had
incurred chained RS failures(receiving RS of a log replay failed again). 
> Since by default, we only allow 3 concurrent SSH handlers(controlled by {code}this.executorService.startExecutorService(ExecutorType.MASTER_SERVER_OPERATIONS,conf.getInt("hbase.master.executor.serverops.threads",
3));{code}).
> If all 3 SSH handlers are doing logReplay(blocking call) and one of receiving RS fails
again then logReplay will hang because regions of the newly failed RS can't be re-assigned
to another live RS(no ssh handler will be processed due to max threads setting) and existing
log replay will keep routing replay traffic to the dead RS.
> The fix is to submit logReplay work into a separate type of executor queue in order not
to block SSH region assignment so that logReplay can route traffic to a live RS after retries
and move forward. 

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

Mime
View raw message