hbase-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "stack (Commented) (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HBASE-5504) Online Merge
Date Thu, 05 Apr 2012 15:48:23 GMT

    [ https://issues.apache.org/jira/browse/HBASE-5504?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13247307#comment-13247307
] 

stack commented on HBASE-5504:
------------------------------

Hey Eric.  Thanks for the help. Looking at FATE it didn't look like we could pull it over
easily (maybe I should look again) but for sure we need to emulate something like it.  First
up would be table read/write lock and have actions like split (or bulk import) take out at
a read lock before progressing.  Can I have a pointer for your file GCer, on why delayed cleanup?
 So you'd keep up in meta the list of files for a range and this would be the region/tablets'
"list" even though the files might not sit physically under a particular region/tablet (What
about compactions?  Would it have to update the list in the meta table when it moved the compacted
file into place?).  On point 1., isn't it possible a file might still over-span a range?

Good stuff.
                
> Online Merge
> ------------
>
>                 Key: HBASE-5504
>                 URL: https://issues.apache.org/jira/browse/HBASE-5504
>             Project: HBase
>          Issue Type: Brainstorming
>          Components: client, master, shell, zookeeper
>    Affects Versions: 0.94.0
>            Reporter: Mubarak Seyed
>             Fix For: 0.96.0
>
>
> As discussed, please refer the discussion at [HBASE-4991|https://issues.apache.org/jira/browse/HBASE-4991]
> Design suggestion from Stack:
> {quote}
> I suggest a design below. It has some prerequisites, some general function that this
feature could use (and others). The prereqs if you think them good, could be done outside
of this JIRA.
> Here's a suggested rough outline of how I think this feature should run. The feature
I'm describing below is merge and deleteRegion for I see them as in essence the same thing.
> (C) Client, (M) Master, RS (Region server), ZK (ZooKeeper)
> 1. Client calls merge or deleteRegion API. API is a range of rows. (C)
> 2. Master gets call. (M)
> 3. Master obtains a write lock on table so it can't be disabled from under us. The write
lock will also disable splitting. This is one of the prereqs I think. Its HBASE-5494 (Or we
could just do something simpler where we have a flag up in zk that splitRegion checks but
thats less useful I think; OR we do the dynamic configs issue and set splits to off via a
config. change). There'd be a timer for how long we wait on the table lock. (M -> ZK)
> 4. If we get the lock, write intent to merge a range up into zk. It also hoists into
zk if its a pure merge or a merge that drops the region data (a deleteRegion call) (M)
> 5. Return to the client either our failed attempt at locking the table or an id of some
sort used identifying this running operation; can use it querying status. (M -> C)
> 6. Turn off balancer. TODO/prereq: Do it in a way that is persisted. Balancer switch
currently in memory only so if master crashes, new master will come up in balancing mode #
(If we had dynamic config. could hoist up to zk a config. that disables the balancer rather
than have a balancer-specific flag/znode OR if a write lock outstanding on a table, then the
balancer does not balance regions in the locked table - this latter might be the easiest to
do) (M)
> 7. Write into zk that just turned off the balancer (If it was on) (M -> ZK)
> 8. Get regions that are involved in the span (M)
> 9. Hoist the list up into zk. (M -> ZK)
> 10. Create region to span the range. (M)
> 11. Write that we did this up into zk. (M -> ZK)
> 12. Close regions in parallel. Confirm close in parallel. (M -> RS)
> 13. Write up into zk regions closed (This might not be necessary since can ask if region
is open). (M -> ZK)
> 14. If a merge and not a delete region, move files under new region. Might multithread
this (moves should go pretty fast). If a deleteregion, we skip this step. (M)
> 15. On completion mark zk (though may not be necessary since its easy to look in fs to
see state of move). (M -> ZK)
> 16. Edit .META. (M)
> 17. Confirm edits went in. (M)
> 18. Move old regions to hbase trash folder TODO: There is no trash folder under /hbase
currently. We should add one. (M)
> 19. Enable balancer (if it was off) (M)
> 20. Unlock table (M)
> {quote}

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

Mime
View raw message