Mailing-List: contact dev-help@hbase.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@hbase.apache.org
MIME-Version: 1.0
In-Reply-To: <CA+RK=_D+9A+a5ojjarhkjkPOmm-eCgwWauMeL6w5qBQfLQFBQQ@mail.gmail.com>
References: <CAAg3a2ovHdekL3ScB2igZV4CnwJ8z9XKGL=mvKTO9wo_PTF+3w@mail.gmail.com>
 <CA+RK=_DVfbzvc9hQw-GUwzJ02t=PAtKETOa_uKpQ5VOVuOm6vA@mail.gmail.com>
 <9085FD13-AD7C-48F1-9ED1-D5F5F5101A38@gmail.com> <CA+RK=_CRcP4CCYijqA574g33kGTaXte53UgpFfwWxzcq+6O5=g@mail.gmail.com>
 <0B4C81CE-B579-4FC5-A015-0253792129A0@gmail.com> <CA+RK=_D+9A+a5ojjarhkjkPOmm-eCgwWauMeL6w5qBQfLQFBQQ@mail.gmail.com>
From: Geoffrey Jacoby <gjacoby@salesforce.com>
Date: Fri, 10 Mar 2017 15:33:46 -0800
Message-ID: <CAE_keTdiMRM=NW6PTn9GqXLDN1JeeoSuOyx5UreOk7VNOqFtnA@mail.gmail.com>
Subject: Re: [VOTE] Backup/Restore feature for HBase 2.0, vote closing 3/11/2017
To: dev@hbase.apache.org
Content-Type: multipart/alternative; boundary=94eb2c115ad208968e054a68cb2c
archived-at: Fri, 10 Mar 2017 23:34:00 -0000

--94eb2c115ad208968e054a68cb2c
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: quoted-printable

I have no vote here, but I'd argue that HBASE-14417 and HBASE-14141
shouldn't be blockers. I agree that HBASE-15227 to add fault tolerance is a
blocker.

HBASE-14417 is support for incrementally backing up bulk loaded rows.
That's an important feature, but if you don't use bulk loads, or don't care
about _incremental_ backups of bulk loads, you'd be able to use backup
quite happily without it. Even if you bulk load occasionally, you can do a
full backup afterward. In the meantime, the docs and help text would just
need to make the limitation clear in big bold letters. :-)

HBASE-14141 is allowing HBase Backup to filter out unnecessary data from
its incremental backups. Since the backup tool allows you to specify that
only certain tables be backed up, incremental backups at WAL granularity
will accidentally backup some rows from unneeded tables. This doesn't
affect the correctness of the to-be-backed-up tables backups or restores,
only its storage cost. The feature works, but there's an important storage
optimization to be done.

As someone who works on clusters that don't use bulk load, and would want
to backup all tables, neither of these seems like a showstopper. However,
if HBASE-14141 would be a breaking change to the backup format, then that
would change my mind about it being a blocker.

Geoffrey


On Fri, Mar 10, 2017 at 3:01 PM, Andrew Purtell <apurtell@apache.org> wrote=
:

> Thanks for the offer but I like that you were honest about compiling a li=
st
> of issues that you thought were blockers for release. Since this proposal
> is a merge into 2.0, and we are trying to release 2.0, I am -1 on this
> merge until those blockers are addressed.
>
> I had a look at the list.
>
> I think the documentation issue is important but not actually a blocker.
> That may be a controversial opinion, but documentation can be back-filled
> worst case. So take HBASE-17133 off the list.
>
> Remaining are effectively HBASE-14417, HBASE-14141, and HBASE-15227. They
> all have patches attached to the respective JIRAs so completing this work
> won't be onerous. Get these committed and I will lift my -1. The others w=
ho
> voted +1 on this thread surely can help with that.
>
> Thanks.
>
>
> On Fri, Mar 10, 2017 at 2:32 PM, Vladimir Rodionov <vladrodionov@gmail.co=
m
> >
> wrote:
>
> > No problem I will downgrade Blockers to Majors if it scares you, Andrew
> =F0=9F=99=82
> >
> > Sent from my iPhone
> >
> > > On Mar 10, 2017, at 1:52 PM, Andrew Purtell <apurtell@apache.org>
> wrote:
> > >
> > > =E2=80=8BI know the merge of this feature has lagged substantially. I=
 think
> that
> > is
> > > regrettable but on another thread we are lamenting that 2.0 is alread=
y
> > > late. Unless I misunderstand, this is a proposal to merge something
> with
> > > known blockers into trunk before we branch it for 2.0 which will
> > > effectively prevent that release because these blockers will be there=
.
> I
> > am
> > > inclined to veto. Probably we should not propose branch merges into
> code
> > we
> > > are trying to get out the door with known blockers. Why not do that
> work
> > > first? It seems an obvious question. Perhaps I am missing something.
> > >
> > > If we can branch for 2.0 now and then merge this, and not into the 2.=
0
> > > branch, I would vote +1 for branch merge even with known blockers
> > pending.
> > > =E2=80=8B
> > >
> > > On Fri, Mar 10, 2017 at 1:42 PM, Vladimir Rodionov <
> > vladrodionov@gmail.com>
> > > wrote:
> > >
> > >> They are not blockers for merge - only for 2.0. GA
> > >> As I said already the feature is usable right now
> > >> We would like to continue working on master and we would like to see=
 a
> > >> commitment from community
> > >>
> > >> Sent from my iPhone
> > >>
> > >> On Mar 10, 2017, at 11:16 AM, Andrew Purtell <apurtell@apache.org>
> > wrote:
> > >>
> > >>>> Only BLOCKERs and CRITICALs are guaranteed for HBase 2.0 release.
> > >>>
> > >>> If we have identified blockers, why merge this before they are in?
> > >>> Otherwise we can't release 2.0, and it is overdue.
> > >>>
> > >>>
> > >>> On Wed, Mar 8, 2017 at 1:32 PM, Vladimir Rodionov <
> > >> vladrodionov@gmail.com>
> > >>> wrote:
> > >>>
> > >>>> Hello, HBase folks
> > >>>>
> > >>>> For your consideration today is Backup/Restore feature for Apache
> > HBAse
> > >>>> 2.0.
> > >>>> Backup code is available as a mega patch in HBASE-14123 (v61),
> applies
> > >>>> cleanly to the current master, all test PASS, patch has no other
> > issues.
> > >>>>
> > >>>> The patch has gone through numerous rounds of code reviews and has
> > >> probably
> > >>>> the most lengthy discussion thread on Apache JIRA (HBASE-14123) :)
> > >>>>
> > >>>> The work has been split into 3 phases (HBASE-14030, 14123, 14414)
> Two
> > >> first
> > >>>> are complete, third one is still in progress.
> > >>>>
> > >>>>
> > >>>> *** Summary of work HBASE-14123
> > >>>>
> > >>>> The new feature introduces new command-line extensions to the hbas=
e
> > >> command
> > >>>> and, from the client side, is accessible through command-line only
> > >>>> Operations:
> > >>>> * Create full backup on a list of tables or backup set
> > >>>> * Create incremental backup image for table list or backup set
> > >>>> * Restore list of tables from a given backup image
> > >>>> * Show current backup progress
> > >>>> * Delete backup image and all related images
> > >>>> * Show history of backups
> > >>>> * Backup set operations: create backup set, add/remove table to/fr=
om
> > >> backup
> > >>>> set, etc
> > >>>>
> > >>>> In the current implementation, the feature is already usable,
> meaning
> > >> that
> > >>>> users can backup tables and restore them using provided command-li=
ne
> > >> tools.
> > >>>> Both: full and incremental backups are supported.
> > >>>> This work is based on original work of IBM team (HBASE-7912). The
> full
> > >> list
> > >>>> of JIRAs included in this mega patch can be found in three umbrell=
a
> > >> JIRAs:
> > >>>> HBASE-14030 (Phase 1), HBASE-14123 (Phase 2) and HBASE-14414 (Phas=
e
> 3
> > -
> > >> all
> > >>>> resolved ones made it into the patch)
> > >>>>
> > >>>> *** What are the remaining work items
> > >>>>
> > >>>> All remaining items can be found in Phase 3 umbrella JIRA:
> > HBASE-14414.
> > >>>> They are split into 3 groups: BLOCKER, CRITICAL, MAJOR
> > >>>> Only BLOCKERs and CRITICALs are guaranteed for HBase 2.0 release.
> > >>>>
> > >>>> ***** BLOCKER
> > >>>>
> > >>>> * HBASE-14417 Incremental backup and bulk loading ( Patch availabl=
e)
> > >>>> * HBASE-14135 HBase Backup/Restore Phase 3: Merge backup images
> > >>>> * HBASE-14141 HBase Backup/Restore Phase 3: Filter WALs on backup =
to
> > >>>> include only edits from backup tables (Patch available)
> > >>>> * HBASE-17133 Backup documentation
> > >>>> * HBASE-15227 Fault tolerance support
> > >>>>
> > >>>> ***** CRITICAL
> > >>>>
> > >>>> * HBASE-16465 Disable split/merges during backup
> > >>>>
> > >>>> We have umbrella JIRA (HBASE-14414) to track all the remaining wor=
k
> > >>>> All the BLOCKER and CRITICAL JIRAs currently in open state will be
> > >>>> implemented by 2.0 release time. Some MAJOR too, but it depends on
> > >> resource
> > >>>> availability
> > >>>> The former development branch (HBASE-7912) is obsolete and will be
> > >>>> closed/deleted after the merge.
> > >>>> We want backup to be a GA feature in 2.0
> > >>>> We are going to support full backward compatibility for backup too=
l
> in
> > >> 2.0
> > >>>> and onwards.
> > >>>>
> > >>>> **** Configuration
> > >>>>
> > >>>> Backup is disabled, by default. To enable it, the following
> > >> configuration
> > >>>> properties must be added to hbase-site.xml:
> > >>>>
> > >>>> hbase.backup.enable=3Dtrue
> > >>>> hbase.master.logcleaner.plugins=3DYOUR_PLUGINS,org.
> > >>>> apache.hadoop.hbase.backup.master.BackupLogCleaner
> > >>>> hbase.procedure.master.classes=3DYOUR_CLASSES,org.
> > >>>> apache.hadoop.hbase.backup.master.LogRollMasterProcedureManager
> > >>>> hbase.procedure.regionserver.classes=3DYOUR_CLASSES,org.
> > >>>> apache.hadoop.hbase.backup.regionserver.
> > LogRollRegionServerProcedureMa
> > >>>> nager
> > >>>>
> > >>>>
> > >>>> I would like to thank IBM team and Jerry He for original work,
> > >>>>
> > >>>> Enis, Ted, Stack, Matteo, Jerry for time spent on code reviews
> > >>>>
> > >>>> Special thanks to Ted Yu for his co-development work.
> > >>>>
> > >>>> References:
> > >>>>
> > >>>> https://issues.apache.org/jira/browse/HBASE-7912 (original IBM,
> > >> contains
> > >>>> design doc)
> > >>>> https://issues.apache.org/jira/browse/HBASE-14030 (Phase 1)
> > >>>> https://issues.apache.org/jira/browse/HBASE-14123 (Phase 2)
> > >>>> https://issues.apache.org/jira/browse/HBASE-14414 (Phase 3)
> > >>>>
> > >>>> Please  vote +1/-1 by midnight Pacific Time (00:00
> > >>>> -0800 GMT) on March 11th  =E2=80=8Bon whether or not we should mer=
ge this
> into
> > >> the
> > >>>> current master.
> > >>>>
> > >>>> -Vladimir Rodionov
> > >>>>
> > >>>
> > >>>
> > >>>
> > >>> --
> > >>> Best regards,
> > >>>
> > >>>  - Andy
> > >>>
> > >>> If you are given a choice, you believe you have acted freely. -
> Raymond
> > >>> Teller (via Peter Watts)
> > >>
> > >
> > >
> > >
> > > --
> > > Best regards,
> > >
> > >   - Andy
> > >
> > > If you are given a choice, you believe you have acted freely. - Raymo=
nd
> > > Teller (via Peter Watts)
> >
>
>
>
> --
> Best regards,
>
>    - Andy
>
> If you are given a choice, you believe you have acted freely. - Raymond
> Teller (via Peter Watts)
>

--94eb2c115ad208968e054a68cb2c--