hbase-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Vladimir Rodionov (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HBASE-7912) HBase Backup/Restore Based on HBase Snapshot
Date Thu, 02 Jul 2015 16:16:07 GMT

    [ https://issues.apache.org/jira/browse/HBASE-7912?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14612142#comment-14612142

Vladimir Rodionov commented on HBASE-7912:

[~mbertozzi] wrote:
you should probably wait or coordinate with the guys over HBASE-13991
I joined the watch list, thanks

Now your questions:
assuming that on the WAL there are multiple regions, is there any logic to decide that using
hfiles instead of wal is better? or do you want always the full ability to restore up to "a
specific seqid"?

First of all, we do not restore to a specific seqId, we restore a specific backup image, which
in turn, for each table is a combination of a last snapshot and sequence of WAL files. WAL
conversion to HFiles is a way to improve restore performance, because HFiles we just copy
over to restore location and WAL files we have to replay.
when do you decide to "compact the wals" on the backup? who is doing that the backup manger?
Its a manual procedure which can be executed by system (global) admin. When to run it is up
to this person to decide.
when you export to another media, who is responsible to compact the wals?
Exporting to another media is the artifact from the original design, feature which won't be
supported. I will update the doc. If by compacting wals you mean - converting them to HFiles
or merging backup images, then this is manual procedures (you will have to run hbase backup
convert/merge tool).
in the document there is mention of "convert the wals to hfiles", in this way we loose the
ability to restore up to "a specific seqid", which goes back to the question, if we don't
need that, is there any logic to decide that using hfiles instead of wal is better for incremental?

Thanks for this question, it turned out that original design has a flaw we have not noticed
before. As since we do not record last backed up seqId and do not restore up to that seqId
- that is kind of tricky imo, there will be some duplicates of KVs in store files after incremental
restore. These duplicates are result of how we do full backup and first incremental backup
after full. 

During full backup we perform distributed log roll and record, for every RS, last WAL timestamp,
then we do snapshot. The next WAL after recorded will make it into next incremental backup
set, but it will contains some edits (puts, deletes) which have been recorded by a previous
snapshot. During restore, we first restore snapshot, then we will replay WALs and create some
duplicates of KVs in different store files. 



> HBase Backup/Restore Based on HBase Snapshot
> --------------------------------------------
>                 Key: HBASE-7912
>                 URL: https://issues.apache.org/jira/browse/HBASE-7912
>             Project: HBase
>          Issue Type: Sub-task
>            Reporter: Richard Ding
>            Assignee: Vladimir Rodionov
>              Labels: backup
>             Fix For: 2.0.0
>         Attachments: HBaseBackupRestore-Jira-7912-DesignDoc-v1.pdf, HBaseBackupRestore-Jira-7912-DesignDoc-v2.pdf,
HBaseBackupRestore-Jira-7912-v4.pdf, HBase_BackupRestore-Jira-7912-CLI-v1.pdf
> Finally, we completed the implementation of our backup/restore solution, and would like
to share with community through this jira. 
> We are leveraging existing hbase snapshot feature, and provide a general solution to
common users. Our full backup is using snapshot to capture metadata locally and using exportsnapshot
to move data to another cluster; the incremental backup is using offline-WALplayer to backup
HLogs; we also leverage global distribution rolllog and flush to improve performance; other
added-on values such as convert, merge, progress report, and CLI commands. So that a common
user can backup hbase data without in-depth knowledge of hbase.  Our solution also contains
some usability features for enterprise users. 
> The detail design document and CLI command will be attached in this jira. We plan to
use 10~12 subtasks to share each of the following features, and document the detail implement
in the subtasks: 
> * *Full Backup* : provide local and remote back/restore for a list of tables
> * *offline-WALPlayer* to convert HLog to HFiles offline (for incremental backup)
> * *distributed* Logroll and distributed flush 
> * Backup *Manifest* and history
> * *Incremental* backup: to build on top of full backup as daily/weekly backup 
> * *Convert*  incremental backup WAL files into hfiles
> * *Merge* several backup images into one(like merge weekly into monthly)
> * *add and remove* table to and from Backup image
> * *Cancel* a backup process
> * backup progress *status*
> * full backup based on *existing snapshot*
> *-------------------------------------------------------------------------------------------------------------*
> *Below is the original description, to keep here as the history for the design and discussion
back in 2013*
> There have been attempts in the past to come up with a viable HBase backup/restore solution
(e.g., HBASE-4618).  Recently, there are many advancements and new features in HBase, for
example, FileLink, Snapshot, and Distributed Barrier Procedure. This is a proposal for a backup/restore
solution that utilizes these new features to achieve better performance and consistency. 
> A common practice of backup and restore in database is to first take full baseline backup,
and then periodically take incremental backup that capture the changes since the full baseline
backup. HBase cluster can store massive amount data.  Combination of full backups with incremental
backups has tremendous benefit for HBase as well.  The following is a typical scenario for
full and incremental backup.
> # The user takes a full backup of a table or a set of tables in HBase. 
> # The user schedules periodical incremental backups to capture the changes from the full
backup, or from last incremental backup.
> # The user needs to restore table data to a past point of time.
> # The full backup is restored to the table(s) or to different table name(s).  Then the
incremental backups that are up to the desired point in time are applied on top of the full
> We would support the following key features and capabilities.
> * Full backup uses HBase snapshot to capture HFiles.
> * Use HBase WALs to capture incremental changes, but we use bulk load of HFiles for fast
incremental restore.
> * Support single table or a set of tables, and column family level backup and restore.
> * Restore to different table names.
> * Support adding additional tables or CF to backup set without interruption of incremental
backup schedule.
> * Support rollup/combining of incremental backups into longer period and bigger incremental
> * Unified command line interface for all the above.
> The solution will support HBase backup to FileSystem, either on the same cluster or across
clusters.  It has the flexibility to support backup to other devices and servers in the future.

This message was sent by Atlassian JIRA

View raw message