hbase-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "jiraposter@reviews.apache.org (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HBASE-5547) Don't delete HFiles when in "backup mode"
Date Thu, 10 May 2012 20:30:51 GMT

    [ https://issues.apache.org/jira/browse/HBASE-5547?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13272727#comment-13272727
] 

jiraposter@reviews.apache.org commented on HBASE-5547:
------------------------------------------------------


-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/4633/
-----------------------------------------------------------

(Updated 2012-05-10 20:28:50.691971)


Review request for hbase, Michael Stack and Lars Hofhansl.


Changes
-------

Refactoring as per stack's comments. Also, updated the testing to avoid flapping. 

I think this is pretty close to ready for commit.


Summary
-------

Essentially, whenever an hfile would be deleted, it is instead moved to the archive directory.
In this impl, the archive directory is on a per table basis, but defaults to '.archive'. Removing
hfiles occurs in three places - compaction, merge and catalog janitor. The former and two
latter are distinctly different code paths, but but did pull out some similarities. The latter
two end up calling the same method, so there should be a reasonable amount of overlap.

Implementation wise: 
    Updated the HMasterInterface to pass the calls onto the zookeeper.
    Added a zk listener to handle updates from the master to the RS to backup.
    Added a utility for removing files and finding archive directories
    Added tests for the regionserver and catalogjanitor approaches.
    Added creation of manager in regionserver.


This addresses bug HBASE-5547.
    https://issues.apache.org/jira/browse/HBASE-5547


Diffs (updated)
-----

  src/main/java/org/apache/hadoop/hbase/HConstants.java 2f432c4 
  src/main/java/org/apache/hadoop/hbase/backup/HFileArchiveCleanup.java PRE-CREATION 
  src/main/java/org/apache/hadoop/hbase/backup/HFileArchiveMonitor.java PRE-CREATION 
  src/main/java/org/apache/hadoop/hbase/backup/HFileArchiveTableMonitor.java PRE-CREATION

  src/main/java/org/apache/hadoop/hbase/backup/HFileDisposer.java PRE-CREATION 
  src/main/java/org/apache/hadoop/hbase/backup/TableHFileArchiveTracker.java PRE-CREATION

  src/main/java/org/apache/hadoop/hbase/client/HBaseAdmin.java 5d4be3f 
  src/main/java/org/apache/hadoop/hbase/client/HFileArchiveManager.java PRE-CREATION 
  src/main/java/org/apache/hadoop/hbase/master/CatalogJanitor.java e751f65 
  src/main/java/org/apache/hadoop/hbase/master/HMaster.java 0ad9b18 
  src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java 1f09be1 
  src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java f7ac81a 
  src/main/java/org/apache/hadoop/hbase/regionserver/RegionServerServices.java 6884d53 
  src/main/java/org/apache/hadoop/hbase/regionserver/Store.java bf1618e 
  src/main/java/org/apache/hadoop/hbase/util/HFileArchiveUtil.java PRE-CREATION 
  src/main/java/org/apache/hadoop/hbase/zookeeper/ZooKeeperWatcher.java 33bc1d0 
  src/main/resources/hbase-default.xml 42d1c4e 
  src/test/java/org/apache/hadoop/hbase/HBaseTestingUtility.java 9fba339 
  src/test/java/org/apache/hadoop/hbase/backup/SimpleHFileArchiveTableMonitor.java PRE-CREATION

  src/test/java/org/apache/hadoop/hbase/backup/TestHFileArchivingCleanup.java PRE-CREATION

  src/test/java/org/apache/hadoop/hbase/backup/TestRegionDisposer.java PRE-CREATION 
  src/test/java/org/apache/hadoop/hbase/master/MockRegionServer.java 69ccc65 
  src/test/java/org/apache/hadoop/hbase/master/TestCatalogJanitor.java 1020374 
  src/test/java/org/apache/hadoop/hbase/regionserver/TestRegionHFileArchiving.java PRE-CREATION

  src/test/java/org/apache/hadoop/hbase/util/HFileArchiveTestingUtil.java PRE-CREATION 
  src/test/java/org/apache/hadoop/hbase/util/MockRegionServerServices.java 3f61cfb 

Diff: https://reviews.apache.org/r/4633/diff


Testing
-------

Added two tests for the separate cases - archiving via the regionserver and for the catalog
tracker. Former runs in a mini cluster and also touches the changes to HMasterInterface and
zookeeper.


Thanks,

Jesse


                
> Don't delete HFiles when in "backup mode"
> -----------------------------------------
>
>                 Key: HBASE-5547
>                 URL: https://issues.apache.org/jira/browse/HBASE-5547
>             Project: HBase
>          Issue Type: New Feature
>            Reporter: Lars Hofhansl
>            Assignee: Jesse Yates
>         Attachments: java_HBASE-5547_v4.patch, java_HBASE-5547_v5.patch, java_HBASE-5547_v6.patch
>
>
> This came up in a discussion I had with Stack.
> It would be nice if HBase could be notified that a backup is in progress (via a znode
for example) and in that case either:
> 1. rename HFiles to be delete to <file>.bck
> 2. rename the HFiles into a special directory
> 3. rename them to a general trash directory (which would not need to be tied to backup
mode).
> That way it should be able to get a consistent backup based on HFiles (HDFS snapshots
or hard links would be better options here, but we do not have those).
> #1 makes cleanup a bit harder.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

Mime
View raw message