hbase-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Jesse Yates (Commented) (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HBASE-5547) Don't delete HFiles when in "backup mode"
Date Wed, 18 Apr 2012 18:56:39 GMT

    [ https://issues.apache.org/jira/browse/HBASE-5547?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13256823#comment-13256823

Jesse Yates commented on HBASE-5547:

Talked offline with Lars about the design of this patch. Currently, it is updating ZK with
the table to archive and then essentially waiting on all the RS to respond that they recieved
the update (simple barrier) before considering backups enabled (in the syncrhonous case).
This means we get high overhead to zk when turning on/off backups, but not a lot when hfiles
are deleted. However, this adds a lot of complexity around watching for nodes, making sure
you get updates, etc. which has been the source of a bit of pain in the development of this

The alternative design is one where the client just writes the 'enable archiving on this table'
node to ZK and then when any of the RS want to delete an hfile they go an read ZK. The thought
here is that we always have a consistent view of which tables need to be archived without
the complexity of keeping up-to-date with the latest state. This means a much simpler design
for checking if we should archive a table and very fast enable/disable for backups.

Clearly, there is a tradeoff in that we have to make a network roundtrip to read if we should
delete or archive a table, but there is little else that needs to happen on the server or
wire (zk holds the info in memory and all thats happening is a checkExists). However, compared
to the time cost in deleting a multi-MB to GB file, this roundtrip seems negligable.

The last alternative would be the dynamic configuration that nicholas recently rolled into
trunk. From what I gather from stack's comments, this would take a disable/enable on a table
for the change to propagate to the HTD (please correct me if I'm wrong here). This just seems
like a lot of flux on a table for a backup situation that I was thinking could occur as often
as a nightly basis. Making the database unavailable just for backup is unreasonable and should
be done dynamically, with minimal impact to the cluster.

What does everyone think? I'm happy to do the rewrite on the patch to make either of the alternative
designs work, as the current on is not quite ready (just needs to have the barrier stuff reworked
slightly to have new/split regions register for  archiving).
> Don't delete HFiles when in "backup mode"
> -----------------------------------------
>                 Key: HBASE-5547
>                 URL: https://issues.apache.org/jira/browse/HBASE-5547
>             Project: HBase
>          Issue Type: New Feature
>            Reporter: Lars Hofhansl
>            Assignee: Jesse Yates
> This came up in a discussion I had with Stack.
> It would be nice if HBase could be notified that a backup is in progress (via a znode
for example) and in that case either:
> 1. rename HFiles to be delete to <file>.bck
> 2. rename the HFiles into a special directory
> 3. rename them to a general trash directory (which would not need to be tied to backup
> That way it should be able to get a consistent backup based on HFiles (HDFS snapshots
or hard links would be better options here, but we do not have those).
> #1 makes cleanup a bit harder.

This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira


View raw message