hbase-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "stack (Commented) (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HBASE-5504) Online Merge
Date Fri, 02 Mar 2012 05:49:02 GMT

    [ https://issues.apache.org/jira/browse/HBASE-5504?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13220689#comment-13220689

stack commented on HBASE-5504:

bq. Since region splitting is disabled after step 3, how do we deal with the case where the
start of row range lies in the middle of a region ?

Good question.  How about, if a merge, we include the hosting region in the merge.  If a delete
region, we throw an exception saying you need to specify region edges.

bq. For step 14, we still need to move data for delete region request because we should have
chosen one of the neighbor regions to cover the hole in .META.

I think this comes of a misunderstanding that you might have Ted.  You can't alter region
edges once created.  For example, the directory in hdfs is the hash of regionname which includes
at least the startkey and should one day include the endkey... If you change the delimiting
keys, you have to make a new region.  Were you thinking we could change the delimiting keys
on an existing region?

On 'What if the master crashes anywhere between step 6 and step 19 ?', its what Mubarak says;
the new master comes up and after initializing, tries to pick up merge/delete from where the
previous master left off... Or simpler, it could just undo it all?

bq. How do we get around if merge/delete-range get stuck (it should not but if it happens???)

I think we need to make the operation cancelable?  In shell/api, there'd be a cancel operation
on table!  (Since you need a write lock to do one of these operations, this would mean one
operation at a time per table only..... maybe no need of there being a lock transaction id
because only one happening at a time?)

Later we can do something better where if an operation does not complete, the master operation
runner would do the cancel.. but that we could do later?

> Online Merge
> ------------
>                 Key: HBASE-5504
>                 URL: https://issues.apache.org/jira/browse/HBASE-5504
>             Project: HBase
>          Issue Type: Brainstorming
>          Components: client, master, shell, zookeeper
>    Affects Versions: 0.94.0
>            Reporter: Mubarak Seyed
>             Fix For: 0.96.0
> As discussed, please refer the discussion at [HBASE-4991|https://issues.apache.org/jira/browse/HBASE-4991]
> Design suggestion from Stack:
> {quote}
> I suggest a design below. It has some prerequisites, some general function that this
feature could use (and others). The prereqs if you think them good, could be done outside
of this JIRA.
> Here's a suggested rough outline of how I think this feature should run. The feature
I'm describing below is merge and deleteRegion for I see them as in essence the same thing.
> (C) Client, (M) Master, RS (Region server), ZK (ZooKeeper)
> 1. Client calls merge or deleteRegion API. API is a range of rows. (C)
> 2. Master gets call. (M)
> 3. Master obtains a write lock on table so it can't be disabled from under us. The write
lock will also disable splitting. This is one of the prereqs I think. Its HBASE-5494 (Or we
could just do something simpler where we have a flag up in zk that splitRegion checks but
thats less useful I think; OR we do the dynamic configs issue and set splits to off via a
config. change). There'd be a timer for how long we wait on the table lock. (M -> ZK)
> 4. If we get the lock, write intent to merge a range up into zk. It also hoists into
zk if its a pure merge or a merge that drops the region data (a deleteRegion call) (M)
> 5. Return to the client either our failed attempt at locking the table or an id of some
sort used identifying this running operation; can use it querying status. (M -> C)
> 6. Turn off balancer. TODO/prereq: Do it in a way that is persisted. Balancer switch
currently in memory only so if master crashes, new master will come up in balancing mode #
(If we had dynamic config. could hoist up to zk a config. that disables the balancer rather
than have a balancer-specific flag/znode OR if a write lock outstanding on a table, then the
balancer does not balance regions in the locked table - this latter might be the easiest to
do) (M)
> 7. Write into zk that just turned off the balancer (If it was on) (M -> ZK)
> 8. Get regions that are involved in the span (M)
> 9. Hoist the list up into zk. (M -> ZK)
> 10. Create region to span the range. (M)
> 11. Write that we did this up into zk. (M -> ZK)
> 12. Close regions in parallel. Confirm close in parallel. (M -> RS)
> 13. Write up into zk regions closed (This might not be necessary since can ask if region
is open). (M -> ZK)
> 14. If a merge and not a delete region, move files under new region. Might multithread
this (moves should go pretty fast). If a deleteregion, we skip this step. (M)
> 15. On completion mark zk (though may not be necessary since its easy to look in fs to
see state of move). (M -> ZK)
> 16. Edit .META. (M)
> 17. Confirm edits went in. (M)
> 18. Move old regions to hbase trash folder TODO: There is no trash folder under /hbase
currently. We should add one. (M)
> 19. Enable balancer (if it was off) (M)
> 20. Unlock table (M)
> {quote}

This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira


View raw message