Return-Path: X-Original-To: apmail-hbase-issues-archive@www.apache.org Delivered-To: apmail-hbase-issues-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id C25EA9625 for ; Wed, 11 Apr 2012 16:49:45 +0000 (UTC) Received: (qmail 42693 invoked by uid 500); 11 Apr 2012 16:49:45 -0000 Delivered-To: apmail-hbase-issues-archive@hbase.apache.org Received: (qmail 42640 invoked by uid 500); 11 Apr 2012 16:49:45 -0000 Mailing-List: contact issues-help@hbase.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Delivered-To: mailing list issues@hbase.apache.org Received: (qmail 42627 invoked by uid 99); 11 Apr 2012 16:49:45 -0000 Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 11 Apr 2012 16:49:45 +0000 X-ASF-Spam-Status: No, hits=-2000.0 required=5.0 tests=ALL_TRUSTED,T_RP_MATCHES_RCVD X-Spam-Check-By: apache.org Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 11 Apr 2012 16:49:41 +0000 Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116]) by hel.zones.apache.org (Postfix) with ESMTP id 3BD0C36591F for ; Wed, 11 Apr 2012 16:49:20 +0000 (UTC) Date: Wed, 11 Apr 2012 16:49:20 +0000 (UTC) From: "jiraposter@reviews.apache.org (Commented) (JIRA)" To: issues@hbase.apache.org Message-ID: <1603119604.13175.1334162960258.JavaMail.tomcat@hel.zones.apache.org> In-Reply-To: <95118742.42325.1331270165325.JavaMail.tomcat@hel.zones.apache.org> Subject: [jira] [Commented] (HBASE-5547) Don't delete HFiles when in "backup mode" MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394 [ https://issues.apache.org/jira/browse/HBASE-5547?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13251727#comment-13251727 ] jiraposter@reviews.apache.org commented on HBASE-5547: ------------------------------------------------------ ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/4633/#review6820 ----------------------------------------------------------- src/main/java/org/apache/hadoop/hbase/client/HBaseAdmin.java done. Originally modeled after the catalog tracker in naming. src/main/java/org/apache/hadoop/hbase/client/HBaseAdmin.java done. src/main/java/org/apache/hadoop/hbase/client/HBaseAdmin.java done. src/main/java/org/apache/hadoop/hbase/client/HBaseAdmin.java done src/main/java/org/apache/hadoop/hbase/client/HBaseAdmin.java done. src/main/java/org/apache/hadoop/hbase/client/HBaseAdmin.java Would rather not, since in that case we would keep around a persistent connection to ZK, which is against what the community has been doing with cutting the client/zk connection (though admittedly that not directly applicable here). src/main/java/org/apache/hadoop/hbase/client/HBaseAdmin.java changed to confirmedArchiving. src/main/java/org/apache/hadoop/hbase/client/HBaseAdmin.java Yeah, it would. But then you should not attempt to expect the archive as valid and fail your backup process. Was going to leave it to the user to determine if they wanted to do the cleanup. However, I would be ok if we added a method that cleaned up the filesystem for files created in the backup after the archiving started. Unfortunately, this means add an extra method on the master which I was trying to avoid, but I guess its not the end of the world. We also have a two-phase commit issue here as we also need to then write a 'consistent-confirmed' file into the archive when it completes to ensure that failed clients don't break the archiving. thoughts? src/main/java/org/apache/hadoop/hbase/client/HBaseAdmin.java done. src/main/java/org/apache/hadoop/hbase/client/HBaseAdmin.java Method name isn't enough? src/main/java/org/apache/hadoop/hbase/client/HFileArchiveManager.java done. src/main/java/org/apache/hadoop/hbase/client/HFileArchiveManager.java done. src/main/java/org/apache/hadoop/hbase/client/HFileArchiveManager.java done - regionsBeingArchived it is. src/main/java/org/apache/hadoop/hbase/regionserver/HFileArchiveMonitor.java done. src/main/java/org/apache/hadoop/hbase/regionserver/HFileArchiveMonitor.java Cruft from the earlier version. src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java yeah, guess it should be. src/main/java/org/apache/hadoop/hbase/util/HFileArchiveUtil.java done. src/main/java/org/apache/hadoop/hbase/util/HFileArchiveUtil.java This is a super rare case where the name of the regiondir to be backed up is the same as the one in the backup already. Just being extra careful that the random storefile name doesn't match, though it is _very_ unlikely to happen, it still could (which could lead to bad circumstances). Currently, the storefile check doesn't check the archive directory and I think it would be pretty big overhead to expect it to do that as well (lots of added dependencies, calls, etc). We can just do the check and renaming here - its not super complicated and seems to work. src/main/java/org/apache/hadoop/hbase/zookeeper/HFileArchiveTracker.java done. src/main/java/org/apache/hadoop/hbase/zookeeper/RegionServerHFileTracker.java done. - Jesse On 2012-04-07 19:51:11, Jesse Yates wrote: bq. bq. ----------------------------------------------------------- bq. This is an automatically generated e-mail. To reply, visit: bq. https://reviews.apache.org/r/4633/ bq. ----------------------------------------------------------- bq. bq. (Updated 2012-04-07 19:51:11) bq. bq. bq. Review request for hbase, Michael Stack and Lars Hofhansl. bq. bq. bq. Summary bq. ------- bq. bq. Essentially, whenever an hfile would be deleted, it is instead moved to the archive directory. In this impl, the archive directory is on a per table basis, but defaults to '.archive'. Removing hfiles occurs in three places - compaction, merge and catalog janitor. The former and two latter are distinctly different code paths, but but did pull out some similarities. The latter two end up calling the same method, so there should be a reasonable amount of overlap. bq. bq. Implementation wise: bq. Updated the HMasterInterface to pass the calls onto the zookeeper. bq. Added a zk listener to handle updates from the master to the RS to backup. bq. Added a utility for removing files and finding archive directories bq. Added tests for the regionserver and catalogjanitor approaches. bq. Added creation of manager in regionserver. bq. bq. bq. This addresses bug HBASE-5547. bq. https://issues.apache.org/jira/browse/HBASE-5547 bq. bq. bq. Diffs bq. ----- bq. bq. src/main/java/org/apache/hadoop/hbase/HConstants.java a4b989e bq. src/main/java/org/apache/hadoop/hbase/client/HBaseAdmin.java 16e4017 bq. src/main/java/org/apache/hadoop/hbase/client/HFileArchiveManager.java PRE-CREATION bq. src/main/java/org/apache/hadoop/hbase/master/CatalogJanitor.java 79d5fdd bq. src/main/java/org/apache/hadoop/hbase/master/HMaster.java fb21bdd bq. src/main/java/org/apache/hadoop/hbase/regionserver/HFileArchiveMonitor.java PRE-CREATION bq. src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java c3df319 bq. src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java 8a61f7d bq. src/main/java/org/apache/hadoop/hbase/regionserver/RegionServerServices.java 6884d53 bq. src/main/java/org/apache/hadoop/hbase/regionserver/Store.java 509a467 bq. src/main/java/org/apache/hadoop/hbase/util/HFileArchiveUtil.java PRE-CREATION bq. src/main/java/org/apache/hadoop/hbase/zookeeper/HFileArchiveTracker.java PRE-CREATION bq. src/main/java/org/apache/hadoop/hbase/zookeeper/RegionServerHFileTracker.java PRE-CREATION bq. src/main/java/org/apache/hadoop/hbase/zookeeper/ZooKeeperWatcher.java 4fc105f bq. src/main/resources/hbase-default.xml 44ee689 bq. src/test/java/org/apache/hadoop/hbase/master/MockRegionServer.java 41616c8 bq. src/test/java/org/apache/hadoop/hbase/master/TestCatalogJanitor.java b4dcb83 bq. src/test/java/org/apache/hadoop/hbase/regionserver/TestRegionHFileArchiving.java PRE-CREATION bq. src/test/java/org/apache/hadoop/hbase/util/HFileArchiveTestingUtil.java PRE-CREATION bq. src/test/java/org/apache/hadoop/hbase/util/MockRegionServerServices.java 7d02759 bq. bq. Diff: https://reviews.apache.org/r/4633/diff bq. bq. bq. Testing bq. ------- bq. bq. Added two tests for the separate cases - archiving via the regionserver and for the catalog tracker. Former runs in a mini cluster and also touches the changes to HMasterInterface and zookeeper. bq. bq. bq. Thanks, bq. bq. Jesse bq. bq. > Don't delete HFiles when in "backup mode" > ----------------------------------------- > > Key: HBASE-5547 > URL: https://issues.apache.org/jira/browse/HBASE-5547 > Project: HBase > Issue Type: New Feature > Reporter: Lars Hofhansl > Assignee: Jesse Yates > > This came up in a discussion I had with Stack. > It would be nice if HBase could be notified that a backup is in progress (via a znode for example) and in that case either: > 1. rename HFiles to be delete to .bck > 2. rename the HFiles into a special directory > 3. rename them to a general trash directory (which would not need to be tied to backup mode). > That way it should be able to get a consistent backup based on HFiles (HDFS snapshots or hard links would be better options here, but we do not have those). > #1 makes cleanup a bit harder. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira