From issues-return-329551-archive-asf-public=cust-asf.ponee.io@hbase.apache.org Fri Jan 12 20:30:08 2018 Return-Path: X-Original-To: archive-asf-public@eu.ponee.io Delivered-To: archive-asf-public@eu.ponee.io Received: from cust-asf.ponee.io (cust-asf.ponee.io [163.172.22.183]) by mx-eu-01.ponee.io (Postfix) with ESMTP id D60E418076D for ; Fri, 12 Jan 2018 20:30:08 +0100 (CET) Received: by cust-asf.ponee.io (Postfix) id C3BB6160C43; Fri, 12 Jan 2018 19:30:08 +0000 (UTC) Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by cust-asf.ponee.io (Postfix) with SMTP id 166B6160C33 for ; Fri, 12 Jan 2018 20:30:07 +0100 (CET) Received: (qmail 16430 invoked by uid 500); 12 Jan 2018 19:30:07 -0000 Mailing-List: contact issues-help@hbase.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Delivered-To: mailing list issues@hbase.apache.org Received: (qmail 16411 invoked by uid 99); 12 Jan 2018 19:30:07 -0000 Received: from pnap-us-west-generic-nat.apache.org (HELO spamd3-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 12 Jan 2018 19:30:07 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd3-us-west.apache.org (ASF Mail Server at spamd3-us-west.apache.org) with ESMTP id 97EC81808E2 for ; Fri, 12 Jan 2018 19:30:06 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd3-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: -107.911 X-Spam-Level: X-Spam-Status: No, score=-107.911 tagged_above=-999 required=6.31 tests=[ENV_AND_HDR_SPF_MATCH=-0.5, KAM_ASCII_DIVIDERS=0.8, RCVD_IN_DNSWL_LOW=-0.7, SPF_PASS=-0.001, T_RP_MATCHES_RCVD=-0.01, USER_IN_DEF_SPF_WL=-7.5, USER_IN_WHITELIST=-100] autolearn=disabled Received: from mx1-lw-eu.apache.org ([10.40.0.8]) by localhost (spamd3-us-west.apache.org [10.40.0.10]) (amavisd-new, port 10024) with ESMTP id Di-GkXqVQmb2 for ; Fri, 12 Jan 2018 19:30:05 +0000 (UTC) Received: from mailrelay1-us-west.apache.org (mailrelay1-us-west.apache.org [209.188.14.139]) by mx1-lw-eu.apache.org (ASF Mail Server at mx1-lw-eu.apache.org) with ESMTP id AC5C15FB71 for ; Fri, 12 Jan 2018 19:30:03 +0000 (UTC) Received: from jira-lw-us.apache.org (unknown [207.244.88.139]) by mailrelay1-us-west.apache.org (ASF Mail Server at mailrelay1-us-west.apache.org) with ESMTP id 7958AE25EB for ; Fri, 12 Jan 2018 19:30:02 +0000 (UTC) Received: from jira-lw-us.apache.org (localhost [127.0.0.1]) by jira-lw-us.apache.org (ASF Mail Server at jira-lw-us.apache.org) with ESMTP id C67C425BE4 for ; Fri, 12 Jan 2018 19:30:01 +0000 (UTC) Date: Fri, 12 Jan 2018 19:30:01 +0000 (UTC) From: "Josh Elser (JIRA)" To: issues@hbase.apache.org Message-ID: In-Reply-To: References: Subject: [jira] [Commented] (HBASE-17852) Add Fault tolerance to HBASE-14417 (Support bulk loaded files in incremental backup) MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394 [ https://issues.apache.org/jira/browse/HBASE-17852?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16324439#comment-16324439 ] Josh Elser commented on HBASE-17852: ------------------------------------ bq. I'd prefer to see this land in master, then we take the concept back to the drawing board and, with all of your help, we revisit this and come up with a design and implementation that works for concurrent backup sessions (as Vlad has this on the Phase4 roadmap already). Ping [~appy]. > Add Fault tolerance to HBASE-14417 (Support bulk loaded files in incremental backup) > ------------------------------------------------------------------------------------ > > Key: HBASE-17852 > URL: https://issues.apache.org/jira/browse/HBASE-17852 > Project: HBase > Issue Type: Sub-task > Reporter: Vladimir Rodionov > Assignee: Vladimir Rodionov > Fix For: 2.0.0 > > Attachments: HBASE-17852-v1.patch, HBASE-17852-v2.patch, HBASE-17852-v3.patch, HBASE-17852-v4.patch, HBASE-17852-v5.patch, HBASE-17852-v6.patch, HBASE-17852-v7.patch, HBASE-17852-v8.patch, HBASE-17852-v9.patch > > > Design approach rollback-via-snapshot implemented in this ticket: > # Before backup create/delete/merge starts we take a snapshot of the backup meta-table (backup system table). This procedure is lightweight because meta table is small, usually should fit a single region. > # When operation fails on a server side, we handle this failure by cleaning up partial data in backup destination, followed by restoring backup meta-table from a snapshot. > # When operation fails on a client side (abnormal termination, for example), next time user will try create/merge/delete he(she) will see error message, that system is in inconsistent state and repair is required, he(she) will need to run backup repair tool. > # To avoid multiple writers to the backup system table (backup client and BackupObserver's) we introduce small table ONLY to keep listing of bulk loaded files. All backup observers will work only with this new tables. The reason: in case of a failure during backup create/delete/merge/restore, when system performs automatic rollback, some data written by backup observers during failed operation may be lost. This is what we try to avoid. > # Second table keeps only bulk load related references. We do not care about consistency of this table, because bulk load is idempotent operation and can be repeated after failure. Partially written data in second table does not affect on BackupHFileCleaner plugin, because this data (list of bulk loaded files) correspond to a files which have not been loaded yet successfully and, hence - are not visible to the system -- This message was sent by Atlassian JIRA (v6.4.14#64029)