hbase-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Ted Yu (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HBASE-14417) Incremental backup and bulk loading
Date Thu, 08 Dec 2016 22:50:58 GMT

    [ https://issues.apache.org/jira/browse/HBASE-14417?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15733635#comment-15733635
] 

Ted Yu commented on HBASE-14417:
--------------------------------

The final filename is available here in HRegion#bulkLoadHFiles() :
{code}
          Path commitedStoreFile = store.bulkLoadHFile(finalPath, seqId);
{code}
If we add one more hook above which records final filename in hbase:backup table, we still
depend on postBulkLoadHFile() hook to write final filename one more time (with state of completion)
- because bulk load event persistence (done in finally block) may fail. Meaning BackupHFileCleaner
wouldn't have enough information whether the bulk load succeeded by simply checking the existence
of store file(s) in region directory:
{code}
        // write a bulk load event when not all hfiles are loaded
        try {
          WALProtos.BulkLoadDescriptor loadDescriptor = ProtobufUtil.toBulkLoadDescriptor(
              this.getRegionInfo().getTable(),
{code}

> Incremental backup and bulk loading
> -----------------------------------
>
>                 Key: HBASE-14417
>                 URL: https://issues.apache.org/jira/browse/HBASE-14417
>             Project: HBase
>          Issue Type: New Feature
>    Affects Versions: 2.0.0
>            Reporter: Vladimir Rodionov
>            Assignee: Ted Yu
>            Priority: Critical
>              Labels: backup
>             Fix For: 2.0.0
>
>         Attachments: 14417-tbl-ext.v10.txt, 14417-tbl-ext.v9.txt, 14417.v1.txt, 14417.v11.txt,
14417.v13.txt, 14417.v2.txt, 14417.v21.txt, 14417.v23.txt, 14417.v24.txt, 14417.v25.txt, 14417.v6.txt
>
>
> Currently, incremental backup is based on WAL files. Bulk data loading bypasses WALs
for obvious reasons, breaking incremental backups. The only way to continue backups after
bulk loading is to create new full backup of a table. This may not be feasible for customers
who do bulk loading regularly (say, every day).
> Google doc for design:
> https://docs.google.com/document/d/1ACCLsecHDvzVSasORgqqRNrloGx4mNYIbvAU7lq5lJE



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message