hbase-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Stack <st...@duboce.net>
Subject Re: [VOTE] Backup/Restore feature for HBase 2.0, vote closing 3/11/2017
Date Thu, 16 Mar 2017 04:17:02 GMT
On Wed, Mar 15, 2017 at 5:51 PM, Ted Yu <yuzhihong@gmail.com> wrote:

> Thanks for the vote, Josh.
>
> So far there have been 3 +1's (Enis', mine and Josh's)
>
> Should we start another thread on the procedure of merging or use this
> thread ?
> Background:
> HBASE-7912 branch is way out of date.
> Latest rounds of mega patch were based on then-current master branch.
>
>
1. Vlad is running this VOTE, not you.
2. Don't you need 3 PMC votes to merge? I count 2.
3. You almost derailed the merge with your earlier interjection rushing to
take us in the wrong direction; you'd think you'd learn from your past
experience but here you are again w/ haste and rash summary.
4. This vote has been awkward. Josh's note is considered and careful. You'd
think it could hang out there a while to see if draws a response rather
than rush to close.

I thought I was reviewing a patch that was against master, not some version
of master from eons ago. Thats a problem. When was the last rebase?

St.Ack





> Thanks
>
> On Wed, Mar 15, 2017 at 4:22 PM, Josh Elser <elserj@apache.org> wrote:
>
> > Spent the day playing around with this on 6 nodes.
> >
> > I've found some rough edges: some known (the docs blocker Vlad pointed
> > out) and some that I think are unknown (tooling is rough -- partially
> from
> > wrong help messages and, I think, changes in design like Master-submitted
> > MR jobs).
> >
> > But, Stack's assessment (and Andrew's reminder) that further tweaking
> > would just throw us back into another review cycle is a real concern.
> It's
> > unfortunate that this feature has lingered so long aside of master to get
> > to this point, but I don't see any realistic resolution to this problem
> > than a merge. In the future, this is something we'll have to try harder
> to
> > avoid letting happen (looking in the mirror with quota work...)
> >
> > Thankfully, I can help out with development/review on the outstanding
> > blockers (notably, the two I pruned from Vlad's original of five -- the
> > other three still seem to be improvements). In addition to these
> blockers,
> > I believe the documentation *must* be updated before a release to note
> that
> > this feature is still growing -- it does not feel like a quality feature
> > that I've come to expect from this community. This isn't a knock on Vlad
> > and company; this is a hard problem and one that I could not have done
> > better given time constraints, but it is also one that users will demand
> > simplicity and the utmost correctness around. To this end, I will also
> try
> > to help out to smooth out these issues in the following 2.x release.
> >
> > So, this leaves me to say: +1 to merge with the caveat that the docs are
> > updated to make sure that any known, user-pitfalls are clearly
> documented.
> > This vote also comes with as real of a promise that I can make to help
> > avoid any issues with this feature that would prevent a 2.0 release.
> >
> > Thanks to all the giants whose shoulders I'm standing on to be able to
> > make this vote.
> >
> > Andrew Purtell wrote:
> >
> >> Great, and I changed my vote to -0 because Stack made a good argument
> that
> >> making more changes would invalidate review up to this point, and I
> trust
> >> this will be resolved before release.
> >>
> >> On Tue, Mar 14, 2017 at 4:29 PM, Josh Elser<elserj@apache.org>  wrote:
> >>
> >> Sorry Andrew, let me clarify as that didn't come out right.
> >>>
> >>> I didn't mean that isn't a conversation worth having _now_, just that I
> >>> was intentionally avoiding it in my previous email because I didn't
> >>> understand the scope of those issues that Vlad had identified. I wanted
> >>> to
> >>> better understand what they were really meant for a user before coming
> to
> >>> my own decision about whether or not I think they are blockers.
> >>>
> >>> That aside, I would agree with you that HBASE-15227 sounds like a real
> >>> blocker. Forcing a new full backup for every table sounds really bad --
> >>> that's not the kind of experience we'd want a user to have.
> >>>
> >>> Andrew Purtell wrote:
> >>>
> >>> I'm going to intentional avoid addressing the discussion of shipping
> >>>> partial features (related, but not relevant at the moment).
> >>>>
> >>>> Then we are not having the same conversation, because it is precisely
> >>>> because this is a vote for this feature to go into 2.0, which is
> already
> >>>> overdue, so should be released yesterday, that I took mention of
> >>>> "blocker"
> >>>> at face value. At least one of them seems to certainly approach this
> >>>> definition. It will be not user friendly, to say the least, to use
> this
> >>>> in
> >>>> a large scale deployment with HBASE-15227 unfinished. HBASE-15227
> >>>> currently
> >>>> has a severity of BLOCKER. Despite what is going on in our politics,
> >>>> words
> >>>> matter and we do not get to redefine them for convenience.
> >>>>
> >>>> Once this work is merged in, how is HBASE-15227 not a blocker for 2.0?
> >>>> Because Vlad offered to reduce its severity to make me feel better?
> >>>> Currently the description on the issue is "System must be tolerant to
> >>>> faults. Backup operations MUST be atomic (no partial completion state
> in
> >>>> system table)" Sounds like a blocker to me, indeed. It is an honest
> >>>> assessment and I don't think anyone is doing the community a favor by
> >>>> trying to walk that back.
> >>>>
> >>>>
> >>>> On Tue, Mar 14, 2017 at 1:57 PM, Josh Elser<elserj@apache.org>
>  wrote:
> >>>>
> >>>> I took a moment to read through the "blockers" as originally
> identified
> >>>> by
> >>>>
> >>>>> Vlad, and (to echo Enis' take) I read the majority of them as being
> >>>>> blockers not for the next release, but for a "full-fledged feature".
> >>>>> I'm
> >>>>> going to intentional avoid addressing the discussion of shipping
> >>>>> partial
> >>>>> features (related, but not relevant at the moment).
> >>>>>
> >>>>> HBASE-15227 is actually the one that bothers me the most, with
> >>>>> HBASE-17133
> >>>>> coming in close behind. Vlad, is there any documentation you can
> point
> >>>>> me
> >>>>> to about what the current issues are with the current implementation?
> >>>>> For
> >>>>> example, what happens now if the system has some kind of "partial
> >>>>> completion state"?
> >>>>>
> >>>>> Documentation is one of those that is really hard to judge. We want
> to
> >>>>> get
> >>>>> this code out for people to use (and to free up our strained dev
> >>>>> resources), but what good is some feature if the docs are
> >>>>> missing/incomplete?
> >>>>>
> >>>>> I think I could stomach the docs being inaccurate (with a clear
> >>>>> disclaimer
> >>>>> that the chapter is incomplete -- that's a 5min task). But, I think I
> >>>>> need
> >>>>> an answer about how the feature handles our common dist-sys category
> of
> >>>>> problems before I can consider whether I'm ok with the feature
> hitting
> >>>>> 2.0...
> >>>>>
> >>>>> I'll also try to throw up a few nodes and play with it to address the
> >>>>> problem as an (ignorant) user ;)
> >>>>>
> >>>>>
> >>>>> Andrew Purtell wrote:
> >>>>>
> >>>>> I don't like that issues were identified as "blockers" but now there
> is
> >>>>>
> >>>>>> an
> >>>>>> attempt to walk that back.
> >>>>>>
> >>>>>> I don't like that development of this feature has lingered for a
> long
> >>>>>> time
> >>>>>> in this unfinished state when this work could have been done by now,
> >>>>>> now
> >>>>>> that we are trying to get a 2.0 out the door. Because this is a
> >>>>>> volunteer
> >>>>>> project I cannot make any demand that it should be done, but I can
> >>>>>> certainly look at the current state and be nonplussed. This will be
> >>>>>> yet
> >>>>>> another half finished thing in 2.0 when this merge happens. Promises
> >>>>>> to
> >>>>>> finish the unfinished work are nice but not currency. Commits are
> >>>>>> currency.
> >>>>>> I hope at least the fault tolerance changes can be completed and
> >>>>>> committed
> >>>>>> before we spin a 2.0 RC, and without causing a 2.0 release to slip
> >>>>>> further.
> >>>>>>
> >>>>>> Also, marking something experimental should be done on the merits of
> >>>>>> that
> >>>>>> evaluation, not simply to justify dropping unfinished work into a
> >>>>>> release
> >>>>>> branch.
> >>>>>>
> >>>>>> I will change my vote to -0.
> >>>>>>
> >>>>>>
> >>>>>> On Mon, Mar 13, 2017 at 4:05 PM, Enis Söztutar<enis@apache.org>
> >>>>>>   wrote:
> >>>>>>
> >>>>>> I think there is some misconception of using the term "blockers" for
> >>>>>>
> >>>>>> referring to those jiras. My understanding is that those three jiras
> >>>>>>> are
> >>>>>>> blockers for the backup functionality to be more mature and more
> >>>>>>> usable.
> >>>>>>> But they are not release blockers. Let's say we merged the code in,
> >>>>>>> and
> >>>>>>> for
> >>>>>>> some reason those did not get addressed in time. We can still do
> the
> >>>>>>> 2.0
> >>>>>>> release without having to wait for the commits. We can instead mark
> >>>>>>> the
> >>>>>>> "backup" feature as experimental with known issues and go on with
> the
> >>>>>>> release. In that sense they are not real release blockers.
> >>>>>>>
> >>>>>>> We are proposing the merge at this time because of the above that
> >>>>>>> maintaining this in a branch is becoming extremely costly and not
> >>>>>>> productive for the HBase community. Realistically, we cannot have
> the
> >>>>>>> luxury of having to wait another couple of months and doing yet
> >>>>>>> another
> >>>>>>> giant round of reviews because the code base is a moving target.
> >>>>>>>
> >>>>>>> Enis
> >>>>>>>
> >>>>>>> On Mon, Mar 13, 2017 at 3:46 PM, Devaraj Das<ddas@hortonworks.com>
> >>>>>>> wrote:
> >>>>>>>
> >>>>>>> Vlad, on the first point, I think what Stack is saying is that
> >>>>>>> creating
> >>>>>>>
> >>>>>>> the new branch (as Ted did) ignores the feedback incorporated thus
> >>>>>>>> far
> >>>>>>>> in
> >>>>>>>> the iterations of the mega-patch. That's a wrong way to go.
> >>>>>>>> On the separation into a backup module, again, that was reverted
> to
> >>>>>>>> ease
> >>>>>>>> reviews of the mega-patch, and was noted as work to be done
> later. I
> >>>>>>>>
> >>>>>>>> think
> >>>>>>>>
> >>>>>>> Stack just wants to make the list of remaining work more complete
> by
> >>>>>>>
> >>>>>>>> citing
> >>>>>>>>
> >>>>>>> that as pending work.
> >>>>>>>
> >>>>>>>> ________________________________________
> >>>>>>>> From: Vladimir Rodionov<vladrodionov@gmail.com>
> >>>>>>>> Sent: Monday, March 13, 2017 3:09 PM
> >>>>>>>> To: dev@hbase.apache.org
> >>>>>>>> Subject: Re: [VOTE] Backup/Restore feature for HBase 2.0, vote
> >>>>>>>> closing
> >>>>>>>> 3/11/2017
> >>>>>>>>
> >>>>>>>>     It ignores the feedback
> >>>>>>>>
> >>>>>>>> If I "ignore" feedback, I put my comment - why? I am always  open
> >>>>>>>>> for
> >>>>>>>>>
> >>>>>>>>> further discussions. If reviewer does not insist on a particular
> >>>>>>>> request
> >>>>>>>>
> >>>>>>>> -
> >>>>>>>>
> >>>>>>> it will be dropped. I think it is fair.
> >>>>>>>
> >>>>>>>> he list is incomplete because a bunch of
> >>>>>>>>
> >>>>>>>> follow-ons came of the review cycle (including moving
> backup/restore
> >>>>>>>>>
> >>>>>>>>>> out
> >>>>>>>>>>
> >>>>>>>>> of
> >>>>>>>>
> >>>>>>>> core to live in its own module).
> >>>>>>>>
> >>>>>>>>> For those who were not following our lengthy conversation on a
> >>>>>>>>> review
> >>>>>>>>>
> >>>>>>>>> board, separation of a backup code into a separate module has
> been
> >>>>>>>> done last year, but has been reverted back by request of a
> reviewer.
> >>>>>>>>
> >>>>>>>>
> >>>>>>>> -Vladimir
> >>>>>>>>
> >>>>>>>> On Mon, Mar 13, 2017 at 2:23 PM, Stack<stack@duboce.net>
> wrote:
> >>>>>>>>
> >>>>>>>> On Fri, Mar 10, 2017 at 9:09 PM, Stack<stack@duboce.net>
> wrote:
> >>>>>>>>
> >>>>>>>> On Fri, Mar 10, 2017 at 6:01 PM, Ted Yu<yuzhihong@gmail.com>
> >>>>>>>>>   wrote:
> >>>>>>>>>
> >>>>>>>>> HBASE-14123 branch has been created, with Vlad's mega patch v61.
> >>>>>>>>>>
> >>>>>>>>>> The patch put up for VOTE here was done on a branch. The call to
> >>>>>>>>>>>
> >>>>>>>>>>> merge
> >>>>>>>>>>
> >>>>>>>>> seems to have been premature given the many cycles of review and
> >>>>>>>> test
> >>>>>>>>
> >>>>>>>> that
> >>>>>>>>>
> >>>>>>>>> happened subsequent (The cycles burned a bunch of dev resource).
> >>>>>>>>>
> >>>>>>>>>> The patch as is is now in a state where it is too big for our
> >>>>>>>>>> infra;
> >>>>>>>>>>
> >>>>>>>>>> rb
> >>>>>>>>>>
> >>>>>>>>> and JIRA are creaking under the size and # of iterations.
> >>>>>>>>
> >>>>>>>> Adding finish of new JIRAs to this merge implies a new round of
> >>>>>>>>>
> >>>>>>>>>> review
> >>>>>>>>>>
> >>>>>>>>> and
> >>>>>>>>
> >>>>>>>> test of an already massive patch. Who is going to do this work?
> >>>>>>>>>
> >>>>>>>>>> Going back to a new branch seems wrong route to take.
> >>>>>>>>>>
> >>>>>>>>>> St.Ack
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>> To be more explicit, this patch was developed on a branch and
> >>>>>>>>>> then a
> >>>>>>>>>>
> >>>>>>>>>> bunch
> >>>>>>>>>
> >>>>>>>> of dev resources were burned getting it into a state where it
> could
> >>>>>>>> be
> >>>>>>>>
> >>>>>>>>> merged to master. Going back to a branch to bulk up the merge so
> it
> >>>>>>>>> includes more JIRAs than the many it already incorporates is the
> >>>>>>>>> wrong
> >>>>>>>>> direction for us to be headed in. It ignores the feedback given
> and
> >>>>>>>>> the
> >>>>>>>>> work done by Vladimir slimming down an already over-broad scope.
> It
> >>>>>>>>> is
> >>>>>>>>>
> >>>>>>>>> also
> >>>>>>>>>
> >>>>>>>> predicated on abundant review and testing resource being on tap to
> >>>>>>>>
> >>>>>>>>> cycle
> >>>>>>>>>
> >>>>>>>> on
> >>>>>>>>
> >>>>>>>> a feature that is useful, but non-core.
> >>>>>>>>
> >>>>>>>>> The patch is ready for merge IMO. Geoffrey makes a nice list of
> >>>>>>>>> what
> >>>>>>>>> is
> >>>>>>>>> still to do though IIRC, the list is incomplete because a bunch
> of
> >>>>>>>>> follow-ons came of the review cycle (including moving
> >>>>>>>>> backup/restore
> >>>>>>>>>
> >>>>>>>>> out
> >>>>>>>>>
> >>>>>>>> of
> >>>>>>>>
> >>>>>>>> core to live in its own module).
> >>>>>>>>
> >>>>>>>>> The patch needs three votes to merge. I am not against merge but
> I
> >>>>>>>>> am
> >>>>>>>>>
> >>>>>>>>> not
> >>>>>>>>>
> >>>>>>>> voting for the patch because I do have any more time to spend on
> >>>>>>>> this
> >>>>>>>>
> >>>>>>>> non-core feature and feel that a vote will have me assume a
> >>>>>>>>>
> >>>>>>>>> responsibility
> >>>>>>>>>
> >>>>>>>> I will not shirk.
> >>>>>>>>
> >>>>>>>>> S
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>> FYI
> >>>>>>>>>
> >>>>>>>>>> On Fri, Mar 10, 2017 at 3:30 PM, Ted Yu<yuzhihong@gmail.com>
> >>>>>>>>>>>
> >>>>>>>>>>> wrote:
> >>>>>>>>>>>
> >>>>>>>>>> Thanks for the feedback, Andrew.
> >>>>>>>>> How about the following plan:
> >>>>>>>>>
> >>>>>>>>>> create branch HBASE-14123 off of master with mega patch v61 as
> the
> >>>>>>>>>>>>
> >>>>>>>>>>>> first
> >>>>>>>>>>>>
> >>>>>>>>>>> commit (reviewed by Stack and Enis)
> >>>>>>>>>>
> >>>>>>>>>> Vlad and I continue development (the 3 blockers) based on
> >>>>>>>>>>>
> >>>>>>>>>>>> HBASE-14123
> >>>>>>>>>>>>
> >>>>>>>>>>>     branch
> >>>>>>>>>> when all of the blockers get +1 and merged into HBASE-14123
> >>>>>>>>>>
> >>>>>>>>>>> branch,
> >>>>>>>>>>>>
> >>>>>>>>>>> we
> >>>>>>>>>>
> >>>>>>>>> propose to community for merging into branch-2 (master branch, if
> >>>>>>>>
> >>>>>>>>> branch-2
> >>>>>>>>>>
> >>>>>>>>>>> doesn't materialize for whatever reason) again
> >>>>>>>>>>>
> >>>>>>>>>>>> Cheers
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>> On Fri, Mar 10, 2017 at 3:01 PM, Andrew Purtell<
> >>>>>>>>>>>>
> >>>>>>>>>>>> apurtell@apache.org>
> >>>>>>>>>>>>
> >>>>>>>>>>> wrote:
> >>>>>>>>>> Thanks for the offer but I like that you were honest about
> >>>>>>>>>>
> >>>>>>>>>>> compiling
> >>>>>>>>>>>>
> >>>>>>>>>>>> a
> >>>>>>>>>>>
> >>>>>>>>>> list
> >>>>>>>>>
> >>>>>>>>>> of issues that you thought were blockers for release. Since this
> >>>>>>>>>>>
> >>>>>>>>>>>> proposal
> >>>>>>>>>>>>>
> >>>>>>>>>>>> is a merge into 2.0, and we are trying to release 2.0, I am -1
> >>>>>>>>>>>> on
> >>>>>>>>>>>> this
> >>>>>>>>>>>>
> >>>>>>>>>>>> merge until those blockers are addressed.
> >>>>>>>>>>> I had a look at the list.
> >>>>>>>>>>>
> >>>>>>>>>>>> I think the documentation issue is important but not actually
> a
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> blocker.
> >>>>>>>>>>>>>
> >>>>>>>>>>>> That may be a controversial opinion, but documentation can be
> >>>>>>>>>>>> back-filled
> >>>>>>>>>>>> worst case. So take HBASE-17133 off the list.
> >>>>>>>>>>>>
> >>>>>>>>>>>> Remaining are effectively HBASE-14417, HBASE-14141, and
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> HBASE-15227.
> >>>>>>>>>>>>>
> >>>>>>>>>>>> They
> >>>>>>>>>>>
> >>>>>>>>>> all have patches attached to the respective JIRAs so completing
> >>>>>>>>>>
> >>>>>>>>>>> this
> >>>>>>>>>>>>
> >>>>>>>>>>>> work
> >>>>>>>>>>>
> >>>>>>>>>> won't be onerous. Get these committed and I will lift my -1. The
> >>>>>>>>>>
> >>>>>>>>>>> others
> >>>>>>>>>>>>
> >>>>>>>>>>>> who
> >>>>>>>>>>> voted +1 on this thread surely can help with that.
> >>>>>>>>>>>
> >>>>>>>>>>>> Thanks.
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> On Fri, Mar 10, 2017 at 2:32 PM, Vladimir Rodionov<
> >>>>>>>>>>>>> vladrodionov@gmail.com>
> >>>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> No problem I will downgrade Blockers to Majors if it scares
> >>>>>>>>>>>>> you,
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> Andrew
> >>>>>>>>>>>>
> >>>>>>>>>>> [image: [image: 🙂]]
> >>>>>>>>>
> >>>>>>>>> Sent from my iPhone
> >>>>>>>>>>>>
> >>>>>>>>>>>>> On Mar 10, 2017, at 1:52 PM, Andrew Purtell<
> >>>>>>>>>>>>>> apurtell@apache.org
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>
> >>>>>>>>>>>> ​I know the merge of this feature has lagged substantially. I
> >>>>>>>>>>
> >>>>>>>>>>> think
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> that
> >>>>>>>>>>>>>
> >>>>>>>>>>>> is
> >>>>>>>>>>>
> >>>>>>>>>>>> regrettable but on another thread we are lamenting that 2.0
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> is
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>> already
> >>>>>>>>>>>>>
> >>>>>>>>>>>> late. Unless I misunderstand, this is a proposal to merge
> >>>>>>>>>
> >>>>>>>>>> something
> >>>>>>>>>>>>> with
> >>>>>>>>>>>>>
> >>>>>>>>>>>> known blockers into trunk before we branch it for 2.0 which
> >>>>>>>>>>>
> >>>>>>>>>>>> will
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> effectively prevent that release because these blockers will
> >>>>>>>>>>>>>
> >>>>>>>>>>>> be
> >>>>>>>>>>
> >>>>>>>>>>> there. I
> >>>>>>>>>>>>>
> >>>>>>>>>>>> am
> >>>>>>>>>
> >>>>>>>>>> inclined to veto. Probably we should not propose branch
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> merges
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>> into
> >>>>>>>>>>>>>
> >>>>>>>>>>>> code
> >>>>>>>>>
> >>>>>>>>>> we
> >>>>>>>>>>>
> >>>>>>>>>>>> are trying to get out the door with known blockers. Why not
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> do
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>> that
> >>>>>>>>>>>>>
> >>>>>>>>>>>> work
> >>>>>>>>>
> >>>>>>>>>> first? It seems an obvious question. Perhaps I am missing
> >>>>>>>>>>>
> >>>>>>>>>>>> something.
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> If we can branch for 2.0 now and then merge this, and not
> >>>>>>>>>>>>> into
> >>>>>>>>>>>>> the
> >>>>>>>>>>>>>
> >>>>>>>>>>>> 2.0
> >>>>>>>>>
> >>>>>>>>>> branch, I would vote +1 for branch merge even with known
> >>>>>>>>>>>
> >>>>>>>>>>>> blockers
> >>>>>>>>>>>>> pending.
> >>>>>>>>>>>>>
> >>>>>>>>>>>> ​
> >>>>>>>>>>
> >>>>>>>>>>> On Fri, Mar 10, 2017 at 1:42 PM, Vladimir Rodionov<
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> vladrodionov@gmail.com>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> They are not blockers for merge - only for 2.0. GA
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> As I said already the feature is usable right now
> >>>>>>>>>>>>>>>> We would like to continue working on master and we would
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> like
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> to
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>> see
> >>>>>>>>
> >>>>>>>>> a
> >>>>>>>>>>
> >>>>>>>>>>> commitment from community
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>> Sent from my iPhone
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> On Mar 10, 2017, at 11:16 AM, Andrew Purtell<
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> apurtell@apache.org
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>> Only BLOCKERs and CRITICALs are guaranteed for HBase 2.0
> >>>>>>>>>>>
> >>>>>>>>>>>> release.
> >>>>>>>>>>>>>>>> If we have identified blockers, why merge this before they
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> are
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>> in?
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>> Otherwise we can't release 2.0, and it is overdue.
> >>>>>>>>>>
> >>>>>>>>>>> On Wed, Mar 8, 2017 at 1:32 PM, Vladimir Rodionov<
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> vladrodionov@gmail.com>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> Hello, HBase folks
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> For your consideration today is Backup/Restore feature
> for
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> Apache
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> HBAse
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> 2.0.
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>> Backup code is available as a mega patch in HBASE-14123
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> (v61),
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> applies
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> cleanly to the current master, all test PASS, patch has no
> >>>>>>>>>>
> >>>>>>>>>>> other
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> issues.
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> The patch has gone through numerous rounds of code reviews
> >>>>>>>>>>>
> >>>>>>>>>>>> and
> >>>>>>>>>>>>>>>> has
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> probably
> >>>>>>>>>>
> >>>>>>>>>>> the most lengthy discussion thread on Apache JIRA
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>> (HBASE-14123)
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> :)
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> The work has been split into 3 phases (HBASE-14030, 14123,
> >>>>>>>>>>>
> >>>>>>>>>>>> 14414)
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>> Two
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> first
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>> are complete, third one is still in progress.
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> *** Summary of work HBASE-14123
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> The new feature introduces new command-line extensions
> to
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> the
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> hbase
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> command
> >>>>>>>>>>
> >>>>>>>>>>> and, from the client side, is accessible through
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>> command-line
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> only
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> Operations:
> >>>>>>>>>>
> >>>>>>>>>>> * Create full backup on a list of tables or backup set
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>> * Create incremental backup image for table list or backup
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> set
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> * Restore list of tables from a given backup image
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> * Show current backup progress
> >>>>>>>>>>
> >>>>>>>>>>> * Delete backup image and all related images
> >>>>>>>>>>>>>>>>>> * Show history of backups
> >>>>>>>>>>>>>>>>>> * Backup set operations: create backup set, add/remove
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> table
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> to/from
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> backup
> >>>>>>>>>
> >>>>>>>>>> set, etc
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> In the current implementation, the feature is already
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> usable,
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> meaning
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> that
> >>>>>>>>>>
> >>>>>>>>>>> users can backup tables and restore them using provided
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> command-line
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> tools.
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> Both: full and incremental backups are supported.
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> This work is based on original work of IBM team
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> (HBASE-7912).
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> The
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> full
> >>>>>>>>>>
> >>>>>>>>>>> list
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>> of JIRAs included in this mega patch can be found in three
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> umbrella
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> JIRAs:
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> HBASE-14030 (Phase 1), HBASE-14123 (Phase 2) and
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>> HBASE-14414
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> (Phase 3
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> -
> >>>>>>>>>
> >>>>>>>>>> all
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> resolved ones made it into the patch)
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> *** What are the remaining work items
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> All remaining items can be found in Phase 3 umbrella
> JIRA:
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> HBASE-14414.
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> They are split into 3 groups: BLOCKER, CRITICAL, MAJOR
> >>>>>>>>>>>>>>>> Only BLOCKERs and CRITICALs are guaranteed for HBase 2.0
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> release.
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> ***** BLOCKER
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> * HBASE-14417 Incremental backup and bulk loading ( Patch
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>> available)
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> * HBASE-14135 HBase Backup/Restore Phase 3: Merge backup
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> images
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> * HBASE-14141 HBase Backup/Restore Phase 3: Filter WALs on
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> backup
> >>>>>>>>>>>
> >>>>>>>>>>>> to
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> include only edits from backup tables (Patch available)
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>> * HBASE-17133 Backup documentation
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> * HBASE-15227 Fault tolerance support
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> ***** CRITICAL
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> * HBASE-16465 Disable split/merges during backup
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> We have umbrella JIRA (HBASE-14414) to track all the
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> remaining
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> work
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> All the BLOCKER and CRITICAL JIRAs currently in open state
> >>>>>>>>>>
> >>>>>>>>>>> will
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>> be
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> implemented by 2.0 release time. Some MAJOR too, but it
> >>>>>>>>>>>
> >>>>>>>>>>>> depends
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>> on
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> resource
> >>>>>>>>>>>
> >>>>>>>>>>>> availability
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>> The former development branch (HBASE-7912) is obsolete and
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> will
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> be
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> closed/deleted after the merge.
> >>>>>>>>>>>
> >>>>>>>>>>>> We want backup to be a GA feature in 2.0
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>> We are going to support full backward compatibility for
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> backup
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> tool in
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> 2.0
> >>>>>>>>>>
> >>>>>>>>>>> and onwards.
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> **** Configuration
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> Backup is disabled, by default. To enable it, the
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> following
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> configuration
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> properties must be added to hbase-site.xml:
> >>>>>>>>>
> >>>>>>>>>> hbase.backup.enable=true
> >>>>>>>>>>>>>>>>>> hbase.master.logcleaner.plugins=YOUR_PLUGINS,org.
> >>>>>>>>>>>>>>>>>> apache.hadoop.hbase.backup.master.BackupLogCleaner
> >>>>>>>>>>>>>>>>>> hbase.procedure.master.classes=YOUR_CLASSES,org.
> >>>>>>>>>>>>>>>>>> apache.hadoop.hbase.backup.master.
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> LogRollMasterProcedureManager
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> hbase.procedure.regionserver.classes=YOUR_CLASSES,org.
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> apache.hadoop.hbase.backup.regionserver.
> >>>>>>>>>>>
> >>>>>>>>>>>> LogRollRegionServerProcedureMa
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> nager
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> I would like to thank IBM team and Jerry He for original
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> work,
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> Enis, Ted, Stack, Matteo, Jerry for time spent on code
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> reviews
> >>>>>>>>>>
> >>>>>>>>>>> Special thanks to Ted Yu for his co-development work.
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> References:
> >>>>>>>>>>
> >>>>>>>>>>> https://issues.apache.org/jira/browse/HBASE-7912
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> (original
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> IBM,
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> contains
> >>>>>>>>>
> >>>>>>>>>> design doc)
> >>>>>>>>>>>
> >>>>>>>>>>>> https://issues.apache.org/jira/browse/HBASE-14030 (Phase
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> 1)
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> https://issues.apache.org/jira/browse/HBASE-14123 (Phase
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> 2)
> >>>>>>>>>
> >>>>>>>>>> https://issues.apache.org/jira/browse/HBASE-14414 (Phase
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> 3)
> >>>>>>>>>
> >>>>>>>>>> Please  vote +1/-1 by midnight Pacific Time (00:00
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> -0800 GMT) on March 11th  ​on whether or not we should
> >>>>>>>>>
> >>>>>>>>>> merge
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> this
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> into
> >>>>>>>>>
> >>>>>>>>>> the
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>> current master.
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> -Vladimir Rodionov
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> --
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> Best regards,
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>     - Andy
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> If you are given a choice, you believe you have acted
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> freely. -
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> Raymond
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>> Teller (via Peter Watts)
> >>>>>>>>>>
> >>>>>>>>>>> --
> >>>>>>>>>>>>>>> Best regards,
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>      - Andy
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> If you are given a choice, you believe you have acted
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> freely. -
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>> Raymond
> >>>>>>>>>>>>>
> >>>>>>>>>>>> Teller (via Peter Watts)
> >>>>>>>>>
> >>>>>>>>>> --
> >>>>>>>>>>>>> Best regards,
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>       - Andy
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> If you are given a choice, you believe you have acted
> freely. -
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> Raymond
> >>>>>>>>>>>>>
> >>>>>>>>>>>> Teller (via Peter Watts)
> >>>>>>>>>>>
> >>>>>>>>>>
> >>>>>>
> >>>>
> >>
> >>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message