hbase-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Stephen Yuan Jiang (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HBASE-14623) Implement dedicated WAL for system tables
Date Tue, 22 Mar 2016 16:45:25 GMT

    [ https://issues.apache.org/jira/browse/HBASE-14623?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15206757#comment-15206757

Stephen Yuan Jiang commented on HBASE-14623:

[~stack], you are right that we need to add tests (and run chaos monkey tests) to see the
benefit of this change before committing the change.  

Since you have some concern on this JIRA, let me give you some background and explanation
(based on customer issue and internal chaos monkey tests):
Today, we have a dedicated WAL for META system table because we don't want to other regions
to wait for META to recovery and want to make sure that META region recovers first.  That
change is about 3 years ago.  Since then, we saw usage of HBASE exploded and we have more
system table that are also very important.  From time to time (in product env), we saw log
split takes long time or even fails due to resource limitation.  If a system table region
is in the part of log in the split (or in the log that are waiting to be split),  the functionality
of the cluster would be affected.  Therefore, we thought of dedicated WAL for system table
approach to speed up system table recovery.  

In terms of benefit to region assignment, we have already had special case for META region
assignment in the Assignment manager code (for user regions, if assignment fails after some
retry, it would give up and queue it and come back to try again later; for meta region, the
retry is forever until assignment succeeds).  From internally chaos money testing, we often
see that a system table region fails after maximum retries (the number is configurable, 10
is the default max retry attempts), without system table regions online, we cannot do a lot
of thing and sometime test fails/times out.  At that time, there is some discussion on whether
to give system table the infinite retry treatment as META region.  When I worked on designing
new proc-v2 based assignment, I exchanged this idea with [~mbertozzi], he had similar thoughts
on this.  We discussed that the recovery for region assignment priority should be META, then
system table, and then user table regions.  When [~tedyu] tried to fix a customer issue in
HBASE-14190 and he had this idea on dedicate system table WAL.  After discussion, we decided
to have this JIRA to separate the work of HBASE-14190 to make it cleaner.  Having dedicated
WAL(s) for system tables would help assign system table region in higher priority than user
table region, and hence speed up system table region assignment.   

Once we agree that there is a need to have dedicated WAL(s) for system tables, then we can
discuss the approach to implement this JIRA.  I think [~tedyu]'s approach is reasonable -
to be safe and minimize the impact on this change, not touch META WAL handling and use a separate
logroller for system table.  However, I am open to more discussion on the approach.  (By the
way, my personal opinion is that we should have sys:meta / sys:namespace / etc. instead of
hbase.meta /hbase:namespace / etc., but too late to make this change :-); the same is that
we have to keep .meta for META table WAL even it is a system table) 

> Implement dedicated WAL for system tables
> -----------------------------------------
>                 Key: HBASE-14623
>                 URL: https://issues.apache.org/jira/browse/HBASE-14623
>             Project: HBase
>          Issue Type: Sub-task
>          Components: wal
>            Reporter: Ted Yu
>            Assignee: Ted Yu
>              Labels: wal
>             Fix For: 2.0.0
>         Attachments: 14623-v1.txt, 14623-v2.txt, 14623-v2.txt, 14623-v2.txt, 14623-v2.txt,
14623-v3.txt, 14623-v4.txt
> As Stephen suggested in parent JIRA, dedicating separate WAL for system tables (other
than hbase:meta) should be done in new JIRA.
> This task is to fulfill the system WAL separation.
> Below is summary of discussion:
> For system table to have its own WAL, we would recover system table faster (fast log
split, fast log replay). It would probably benefit 
> AssignmentManager on system table region assignment. At this time, the new AssignmentManager
is not planned to change WAL. So the existence of this JIRA is good for overall system, not
specific to AssignmentManager.
> There are 3 strategies for implementing system table WAL:
> 1. one WAL for all non-meta system tables
> 2. one WAL for each non-meta system table
> 3. one WAL for each region of non-meta system table
> Currently most system tables are one region table (only ACL table may become big). Choices
2 and 3 basically are the same.
> From implementation point of view, choices 2 and 3 are cleaner than choice 1 (as we have
already had 1 WAL for META table and we can reuse the logic). With choice 2 or 3, assignment
manager performance should not be impacted and it would be easier for assignment manager to
assign system table region (eg. without waiting for user table log split to complete for assigning
system table region).

This message was sent by Atlassian JIRA

View raw message