hbase-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "nkeywal (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HBASE-5843) Improve HBase MTTR - Mean Time To Recover
Date Fri, 07 Sep 2012 13:54:08 GMT

    [ https://issues.apache.org/jira/browse/HBASE-5843?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13450625#comment-13450625

nkeywal commented on HBASE-5843:

Some tests and analysis around Distributed Split

Scenario with a single machine, killing only a region server
- dfs.replication = 2
- local HD. The test failed with the ramDrive. 
- Start with 2 DN and 2 RS. Create a table with 100 regions in the second one. The first holds
meta & root.
- Insert 10M rows, distributed on all regions. That creates 8 logs files of 60Mb each.
- start 3 more DN & RS.
- kill -9 the second RS (NOT the datanode).
- Wait for the regions to be available again.

~180s detection time (sometimes less, it's strange. But always more than 120s).
~25s split (two tasks of ~10s each per regionserver)
~30s assignment

~0s detection time (because it's a kill -9, so we have HBASE-5844. A hw failure would bring
the same result as 0.94).
~25s split (two tasks of ~10s each)
~30s assignment. Other tests seems to show that the actual time is taken on replaying the

=> No difference in time here. Except the detection time, that will not play a role in
a HW failure.
=> A split task in done in 10s. If you have a reasonable cluster, you can expect this task
to takes 10s in production. 
=> Same should apply to replaying edits, if the regions are well redistributed among the
other machines.
=> Still, the assignment/replaying could be faster. Analysis to do, and JIRA to create.

As of today, if the regionserver crashes, we should have, in production:
- 0.94: ~50s (30s detection + 10s split + 10s assignment)
- 0.96: ~20s (0s detection + 10s split + 10s assignment)

If it's a HW failure, we need to take into account that we've lost a datanode as well.
> Improve HBase MTTR - Mean Time To Recover
> -----------------------------------------
>                 Key: HBASE-5843
>                 URL: https://issues.apache.org/jira/browse/HBASE-5843
>             Project: HBase
>          Issue Type: Umbrella
>    Affects Versions: 0.96.0
>            Reporter: nkeywal
>            Assignee: nkeywal
> A part of the approach is described here: https://docs.google.com/document/d/1z03xRoZrIJmg7jsWuyKYl6zNournF_7ZHzdi0qz_B4c/edit
> The ideal target is:
> - failure impact client applications only by an added delay to execute a query, whatever
the failure.
> - this delay is always inferior to 1 second.
> We're not going to achieve that immediately...
> Priority will be given to the most frequent issues.
> Short term:
> - software crash
> - standard administrative tasks as stop/start of a cluster.

This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

View raw message