hbase-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Hudson (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HBASE-14417) Incremental backup and bulk loading
Date Wed, 29 Mar 2017 03:28:41 GMT

    [ https://issues.apache.org/jira/browse/HBASE-14417?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15946469#comment-15946469
] 

Hudson commented on HBASE-14417:
--------------------------------

FAILURE: Integrated in Jenkins build HBase-Trunk_matrix #2758 (See [https://builds.apache.org/job/HBase-Trunk_matrix/2758/])
HBASE-14417 Incremental backup and bulk loading (tedyu: rev 0345fc87759a7d44ecc385327ebb586fc632fb65)
* (add) hbase-server/src/main/java/org/apache/hadoop/hbase/backup/BackupObserver.java
* (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/backup/impl/BackupManager.java
* (add) hbase-server/src/test/java/org/apache/hadoop/hbase/backup/TestIncrementalBackupWithBulkLoad.java
* (edit) hbase-server/src/test/java/org/apache/hadoop/hbase/backup/TestBackupBase.java
* (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/backup/impl/BackupAdminImpl.java
* (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/backup/impl/RestoreTablesClient.java
* (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/mapreduce/LoadIncrementalHFiles.java
* (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/util/HFileArchiveUtil.java
* (add) hbase-server/src/test/java/org/apache/hadoop/hbase/backup/TestBackupHFileCleaner.java
* (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/backup/mapreduce/MapReduceRestoreJob.java
* (edit) hbase-server/src/test/java/org/apache/hadoop/hbase/mapreduce/TestLoadIncrementalHFiles.java
* (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/backup/impl/IncrementalTableBackupClient.java
* (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/backup/impl/BackupSystemTable.java
* (add) hbase-server/src/main/java/org/apache/hadoop/hbase/backup/BackupHFileCleaner.java


> Incremental backup and bulk loading
> -----------------------------------
>
>                 Key: HBASE-14417
>                 URL: https://issues.apache.org/jira/browse/HBASE-14417
>             Project: HBase
>          Issue Type: New Feature
>            Reporter: Vladimir Rodionov
>            Assignee: Ted Yu
>            Priority: Blocker
>              Labels: backup
>             Fix For: 2.0
>
>         Attachments: 14417-tbl-ext.v10.txt, 14417-tbl-ext.v11.txt, 14417-tbl-ext.v14.txt,
14417-tbl-ext.v18.txt, 14417-tbl-ext.v19.txt, 14417-tbl-ext.v20.txt, 14417-tbl-ext.v21.txt,
14417-tbl-ext.v22.txt, 14417-tbl-ext.v23.txt, 14417-tbl-ext.v24.txt, 14417-tbl-ext.v9.txt,
14417.v11.txt, 14417.v13.txt, 14417.v1.txt, 14417.v21.txt, 14417.v23.txt, 14417.v24.txt, 14417.v25.txt,
14417.v2.txt, 14417.v6.txt
>
>
> Currently, incremental backup is based on WAL files. Bulk data loading bypasses WALs
for obvious reasons, breaking incremental backups. The only way to continue backups after
bulk loading is to create new full backup of a table. This may not be feasible for customers
who do bulk loading regularly (say, every day).
> Here is the review board (out of date):
> https://reviews.apache.org/r/54258/
> In order not to miss the hfiles which are loaded into region directories in a situation
where postBulkLoadHFile() hook is not called (bulk load being interrupted), we record hfile
names thru preCommitStoreFile() hook.
> At time of incremental backup, we check the presence of such hfiles. If they are present,
they become part of the incremental backup image.
> Here is review board:
> https://reviews.apache.org/r/57790/
> Google doc for design:
> https://docs.google.com/document/d/1ACCLsecHDvzVSasORgqqRNrloGx4mNYIbvAU7lq5lJE



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

Mime
View raw message