Mailing-List: contact issues-help@hbase.apache.org; run by ezmlm
Precedence: bulk
Date: Thu, 30 Nov 2017 20:08:00 +0000 (UTC)
From: "Vladimir Rodionov (JIRA)" <jira@apache.org>
To: issues@hbase.apache.org
Message-ID: <JIRA.13060154.1490821740000.358582.1512072480415@Atlassian.JIRA>
In-Reply-To: <JIRA.13060154.1490821740000@Atlassian.JIRA>
References: <JIRA.13060154.1490821740000@Atlassian.JIRA> <JIRA.13060154.1490821740887@jira-lw-us.apache.org>
Subject: [jira] [Commented] (HBASE-17852) Add Fault tolerance to HBASE-14417
 (Support bulk loaded files in incremental backup)
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
archived-at: Thu, 30 Nov 2017 20:08:05 -0000


    [ https://issues.apache.org/jira/browse/HBASE-17852?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16273329#comment-16273329 ] 

Vladimir Rodionov commented on HBASE-17852:
-------------------------------------------

The reason is the simplicity of the implementation. Is not this obvious? Should I have spent time trying to implement Tx management instead? I doubt. Did I answer original question? I thought that we are technical guys and we need technical answers. It seems that I was wrong. 

User intervention is required only if user kills backup process or it dies on a client side, for some other reason. All cluster side failures get repaired automatically. I see nothing painful for users here, [~mdrob], especially when I will implement auto-repair feature. This is the question actually, should we do repair automatically or we need to inform user, that there was abnormal failure of a last backup/merge/delete command and user need to run repair.

Do not we still have *hbck* for this reason? Repair all the s**t which happens periodically in HBase cluster.

Moving feature out of beta-1 only because someone does not like *attitude of a contributor* means that something is not going well in HBase community.  

> Add Fault tolerance to HBASE-14417 (Support bulk loaded files in incremental backup)
> ------------------------------------------------------------------------------------
>
>                 Key: HBASE-17852
>                 URL: https://issues.apache.org/jira/browse/HBASE-17852
>             Project: HBase
>          Issue Type: Sub-task
>            Reporter: Vladimir Rodionov
>            Assignee: Vladimir Rodionov
>             Fix For: 2.0.0
>
>         Attachments: HBASE-17852-v1.patch, HBASE-17852-v2.patch, HBASE-17852-v3.patch, HBASE-17852-v4.patch, HBASE-17852-v5.patch, HBASE-17852-v6.patch, HBASE-17852-v7.patch, HBASE-17852-v8.patch, HBASE-17852-v9.patch
>
>
> Design approach rollback-via-snapshot implemented in this ticket:
> # Before backup create/delete/merge starts we take a snapshot of the backup meta-table (backup system table). This procedure is lightweight because meta table is small, usually should fit a single region.
> # When operation fails on a server side, we handle this failure by cleaning up partial data in backup destination, followed by restoring backup meta-table from a snapshot. 
> # When operation fails on a client side (abnormal termination, for example), next time user will try create/merge/delete he(she) will see error message, that system is in inconsistent state and repair is required, he(she) will need to run backup repair tool.
> # To avoid multiple writers to the backup system table (backup client and BackupObserver's) we introduce small table ONLY to keep listing of bulk loaded files. All backup observers will work only with this new tables. The reason: in case of a failure during backup create/delete/merge/restore, when system performs automatic rollback, some data written by backup observers during failed operation may be lost. This is what we try to avoid.
> # Second table keeps only bulk load related references. We do not care about consistency of this table, because bulk load is idempotent operation and can be repeated after failure. Partially written data in second table does not affect on BackupHFileCleaner plugin, because this data (list of bulk loaded files) correspond to a files which have not been loaded yet successfully and, hence - are not visible to the system 


--
This message was sent by Atlassian JIRA
(v6.4.14#64029)