hbase-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "jiraposter@reviews.apache.org (Commented) (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HBASE-5547) Don't delete HFiles when in "backup mode"
Date Thu, 05 Apr 2012 18:42:33 GMT

    [ https://issues.apache.org/jira/browse/HBASE-5547?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13247503#comment-13247503
] 

jiraposter@reviews.apache.org commented on HBASE-5547:
------------------------------------------------------


-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/4633/#review6714
-----------------------------------------------------------


Thanks for the review Ted. Any thoughts on combining the Tracker and Manager (see inline comment
below on HFileArchiveManager).


src/main/java/org/apache/hadoop/hbase/HConstants.java
<https://reviews.apache.org/r/4633/#comment14522>

    fair enough. done.



src/main/java/org/apache/hadoop/hbase/ipc/HMasterInterface.java
<https://reviews.apache.org/r/4633/#comment14517>

    Disable right now just wipes out all the tables currently in backup mode (not saving any
info), but I guess it could be reasonable to want to start backing up all the tables - though
in real usage you probably only want to do one table at a time to minimize impact on a cluster.

    
    I'd rather not give people that power directly, but make them jump through a couple hoops
(listing out all the tables and then iteratively starting backups on those tables) due: 1)
how much data that can be, and 2) in a real 'backup' context this means they are cloning all
the tables (doing a full table scan on all tables) which would be highly disruptive.



src/main/java/org/apache/hadoop/hbase/regionserver/HFileArchiveManager.java
<https://reviews.apache.org/r/4633/#comment14518>

    done.



src/main/java/org/apache/hadoop/hbase/regionserver/HFileArchiveManager.java
<https://reviews.apache.org/r/4633/#comment14519>

    done



src/main/java/org/apache/hadoop/hbase/regionserver/HFileArchiveManager.java
<https://reviews.apache.org/r/4633/#comment14520>

    I've been debating whether or not to roll this implementation into the HFileArchiveTracker
and just move this into an interface. It doesn't actually do very much - essentially a prettier
front to a map and could easily be moved into the tracker. 
    
    Thoughts?



src/main/java/org/apache/hadoop/hbase/regionserver/HFileArchiveManager.java
<https://reviews.apache.org/r/4633/#comment14521>

    done.



src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java
<https://reviews.apache.org/r/4633/#comment14526>

    done.



src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java
<https://reviews.apache.org/r/4633/#comment14523>

    That case should only come up in testing, so logging seems excessive.



src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java
<https://reviews.apache.org/r/4633/#comment14524>

    used store because in code that's what we are really dealing with, but everyone just refers
to them as hfiles. going with your way for consistency.



src/main/java/org/apache/hadoop/hbase/regionserver/RegionServerServices.java
<https://reviews.apache.org/r/4633/#comment14525>

    done.



src/main/java/org/apache/hadoop/hbase/util/HFileArchiveUtil.java
<https://reviews.apache.org/r/4633/#comment14527>

    done.



src/main/java/org/apache/hadoop/hbase/util/HFileArchiveUtil.java
<https://reviews.apache.org/r/4633/#comment14528>

    yup, nice catch (no pun intended).



src/main/java/org/apache/hadoop/hbase/util/HFileArchiveUtil.java
<https://reviews.apache.org/r/4633/#comment14529>

    true. earlier iteration comment that got confused with manager.keepFiles



src/main/java/org/apache/hadoop/hbase/util/HFileArchiveUtil.java
<https://reviews.apache.org/r/4633/#comment14530>

    how so? renames an the src path to the dest path. Seems to work in tests.



src/main/java/org/apache/hadoop/hbase/util/HFileArchiveUtil.java
<https://reviews.apache.org/r/4633/#comment14531>

    then its something we don't care about - only care about the hfiles, which are stored
in one directory level below the region.



src/main/java/org/apache/hadoop/hbase/util/HFileArchiveUtil.java
<https://reviews.apache.org/r/4633/#comment14532>

    done.



src/main/java/org/apache/hadoop/hbase/util/HFileArchiveUtil.java
<https://reviews.apache.org/r/4633/#comment14534>

    Definitely need to add another unit test covering the 'resolve' part of this method -
lot of gymnastics/corner cases going on.



src/main/java/org/apache/hadoop/hbase/util/HFileArchiveUtil.java
<https://reviews.apache.org/r/4633/#comment14536>

    reworked a little bit in the new version.



src/main/java/org/apache/hadoop/hbase/util/HFileArchiveUtil.java
<https://reviews.apache.org/r/4633/#comment14533>

    going to bail out earlier if we can't delete the old file.



src/main/java/org/apache/hadoop/hbase/util/HFileArchiveUtil.java
<https://reviews.apache.org/r/4633/#comment14535>

    done.



src/main/java/org/apache/hadoop/hbase/zookeeper/HFileArchiveTracker.java
<https://reviews.apache.org/r/4633/#comment14537>

    done.



src/main/java/org/apache/hadoop/hbase/zookeeper/HFileArchiveTracker.java
<https://reviews.apache.org/r/4633/#comment14538>

    don't think so - enabled only means that it found the archive znode, in which case it
will attempt to update the manager. The enabled construct is really an internal util, as opposed
to something that anyone else should really care about (at least that was the intention.)



src/main/java/org/apache/hadoop/hbase/zookeeper/HFileArchiveTracker.java
<https://reviews.apache.org/r/4633/#comment14539>

    oops, shouldn't have. Adding unit test to catch that.


- Jesse


On 2012-04-04 01:23:29, Jesse Yates wrote:
bq.  
bq.  -----------------------------------------------------------
bq.  This is an automatically generated e-mail. To reply, visit:
bq.  https://reviews.apache.org/r/4633/
bq.  -----------------------------------------------------------
bq.  
bq.  (Updated 2012-04-04 01:23:29)
bq.  
bq.  
bq.  Review request for hbase, Michael Stack and Lars Hofhansl.
bq.  
bq.  
bq.  Summary
bq.  -------
bq.  
bq.  Essentially, whenever an hfile would be deleted, it is instead moved to the archive directory.
In this impl, the archive directory is on a per table basis, but defaults to '.archive'. Removing
hfiles occurs in three places - compaction, merge and catalog janitor. The former and two
latter are distinctly different code paths, but but did pull out some similarities. The latter
two end up calling the same method, so there should be a reasonable amount of overlap.
bq.  
bq.  Implementation wise: 
bq.      Updated the HMasterInterface to pass the calls onto the zookeeper.
bq.      Added a zk listener to handle updates from the master to the RS to backup.
bq.      Added a utility for removing files and finding archive directories
bq.      Added tests for the regionserver and catalogjanitor approaches.
bq.      Added creation of manager in regionserver.
bq.  
bq.  
bq.  This addresses bug HBASE-5547.
bq.      https://issues.apache.org/jira/browse/HBASE-5547
bq.  
bq.  
bq.  Diffs
bq.  -----
bq.  
bq.    src/main/java/org/apache/hadoop/hbase/HConstants.java 21ac4ba 
bq.    src/main/java/org/apache/hadoop/hbase/client/HBaseAdmin.java 16e4017 
bq.    src/main/java/org/apache/hadoop/hbase/ipc/HMasterInterface.java ea7ae45 
bq.    src/main/java/org/apache/hadoop/hbase/master/CatalogJanitor.java 79d5fdd 
bq.    src/main/java/org/apache/hadoop/hbase/master/HMaster.java 9bd4ace 
bq.    src/main/java/org/apache/hadoop/hbase/regionserver/HFileArchiveManager.java PRE-CREATION

bq.    src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java 1d11f8f 
bq.    src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java 4f80999 
bq.    src/main/java/org/apache/hadoop/hbase/regionserver/RegionServerServices.java 6884d53

bq.    src/main/java/org/apache/hadoop/hbase/regionserver/Store.java 0c7b396 
bq.    src/main/java/org/apache/hadoop/hbase/util/HFileArchiveUtil.java PRE-CREATION 
bq.    src/main/java/org/apache/hadoop/hbase/zookeeper/HFileArchiveTracker.java PRE-CREATION

bq.    src/main/java/org/apache/hadoop/hbase/zookeeper/ZooKeeperWatcher.java 0f83655 
bq.    src/main/resources/hbase-default.xml 341431a 
bq.    src/test/java/org/apache/hadoop/hbase/master/MockRegionServer.java d2b3060 
bq.    src/test/java/org/apache/hadoop/hbase/master/TestCatalogJanitor.java 227c5f2 
bq.    src/test/java/org/apache/hadoop/hbase/regionserver/TestRegionHFileArchiving.java PRE-CREATION

bq.    src/test/java/org/apache/hadoop/hbase/util/MockRegionServerServices.java bb3ddd7 
bq.  
bq.  Diff: https://reviews.apache.org/r/4633/diff
bq.  
bq.  
bq.  Testing
bq.  -------
bq.  
bq.  Added two tests for the separate cases - archiving via the regionserver and for the catalog
tracker. Former runs in a mini cluster and also touches the changes to HMasterInterface and
zookeeper.
bq.  
bq.  
bq.  Thanks,
bq.  
bq.  Jesse
bq.  
bq.


                
> Don't delete HFiles when in "backup mode"
> -----------------------------------------
>
>                 Key: HBASE-5547
>                 URL: https://issues.apache.org/jira/browse/HBASE-5547
>             Project: HBase
>          Issue Type: New Feature
>            Reporter: Lars Hofhansl
>            Assignee: Jesse Yates
>
> This came up in a discussion I had with Stack.
> It would be nice if HBase could be notified that a backup is in progress (via a znode
for example) and in that case either:
> 1. rename HFiles to be delete to <file>.bck
> 2. rename the HFiles into a special directory
> 3. rename them to a general trash directory (which would not need to be tied to backup
mode).
> That way it should be able to get a consistent backup based on HFiles (HDFS snapshots
or hard links would be better options here, but we do not have those).
> #1 makes cleanup a bit harder.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

Mime
View raw message