hbase-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ted Yu <yuzhih...@gmail.com>
Subject Re: [VOTE] Backup/Restore feature for HBase 2.0, vote closing 3/11/2017
Date Thu, 16 Mar 2017 02:34:38 GMT
Clarification:
Due to the divergence of HBASE-7912 branch from master branch, I think the
conclusion of this vote means that mega patch v61 on HBASE-14123 would be
integrated to master branch directly (with credit to people who
contributed).

On Wed, Mar 15, 2017 at 5:51 PM, Ted Yu <yuzhihong@gmail.com> wrote:

> Thanks for the vote, Josh.
>
> So far there have been 3 +1's (Enis', mine and Josh's)
>
> Should we start another thread on the procedure of merging or use this
> thread ?
> Background:
> HBASE-7912 branch is way out of date.
> Latest rounds of mega patch were based on then-current master branch.
>
> Thanks
>
> On Wed, Mar 15, 2017 at 4:22 PM, Josh Elser <elserj@apache.org> wrote:
>
>> Spent the day playing around with this on 6 nodes.
>>
>> I've found some rough edges: some known (the docs blocker Vlad pointed
>> out) and some that I think are unknown (tooling is rough -- partially from
>> wrong help messages and, I think, changes in design like Master-submitted
>> MR jobs).
>>
>> But, Stack's assessment (and Andrew's reminder) that further tweaking
>> would just throw us back into another review cycle is a real concern. It's
>> unfortunate that this feature has lingered so long aside of master to get
>> to this point, but I don't see any realistic resolution to this problem
>> than a merge. In the future, this is something we'll have to try harder to
>> avoid letting happen (looking in the mirror with quota work...)
>>
>> Thankfully, I can help out with development/review on the outstanding
>> blockers (notably, the two I pruned from Vlad's original of five -- the
>> other three still seem to be improvements). In addition to these blockers,
>> I believe the documentation *must* be updated before a release to note that
>> this feature is still growing -- it does not feel like a quality feature
>> that I've come to expect from this community. This isn't a knock on Vlad
>> and company; this is a hard problem and one that I could not have done
>> better given time constraints, but it is also one that users will demand
>> simplicity and the utmost correctness around. To this end, I will also try
>> to help out to smooth out these issues in the following 2.x release.
>>
>> So, this leaves me to say: +1 to merge with the caveat that the docs are
>> updated to make sure that any known, user-pitfalls are clearly documented.
>> This vote also comes with as real of a promise that I can make to help
>> avoid any issues with this feature that would prevent a 2.0 release.
>>
>> Thanks to all the giants whose shoulders I'm standing on to be able to
>> make this vote.
>>
>> Andrew Purtell wrote:
>>
>>> Great, and I changed my vote to -0 because Stack made a good argument
>>> that
>>> making more changes would invalidate review up to this point, and I trust
>>> this will be resolved before release.
>>>
>>> On Tue, Mar 14, 2017 at 4:29 PM, Josh Elser<elserj@apache.org>  wrote:
>>>
>>> Sorry Andrew, let me clarify as that didn't come out right.
>>>>
>>>> I didn't mean that isn't a conversation worth having _now_, just that I
>>>> was intentionally avoiding it in my previous email because I didn't
>>>> understand the scope of those issues that Vlad had identified. I wanted
>>>> to
>>>> better understand what they were really meant for a user before coming
>>>> to
>>>> my own decision about whether or not I think they are blockers.
>>>>
>>>> That aside, I would agree with you that HBASE-15227 sounds like a real
>>>> blocker. Forcing a new full backup for every table sounds really bad --
>>>> that's not the kind of experience we'd want a user to have.
>>>>
>>>> Andrew Purtell wrote:
>>>>
>>>> I'm going to intentional avoid addressing the discussion of shipping
>>>>> partial features (related, but not relevant at the moment).
>>>>>
>>>>> Then we are not having the same conversation, because it is precisely
>>>>> because this is a vote for this feature to go into 2.0, which is
>>>>> already
>>>>> overdue, so should be released yesterday, that I took mention of
>>>>> "blocker"
>>>>> at face value. At least one of them seems to certainly approach this
>>>>> definition. It will be not user friendly, to say the least, to use
>>>>> this in
>>>>> a large scale deployment with HBASE-15227 unfinished. HBASE-15227
>>>>> currently
>>>>> has a severity of BLOCKER. Despite what is going on in our politics,
>>>>> words
>>>>> matter and we do not get to redefine them for convenience.
>>>>>
>>>>> Once this work is merged in, how is HBASE-15227 not a blocker for 2.0?
>>>>> Because Vlad offered to reduce its severity to make me feel better?
>>>>> Currently the description on the issue is "System must be tolerant to
>>>>> faults. Backup operations MUST be atomic (no partial completion state
>>>>> in
>>>>> system table)" Sounds like a blocker to me, indeed. It is an honest
>>>>> assessment and I don't think anyone is doing the community a favor by
>>>>> trying to walk that back.
>>>>>
>>>>>
>>>>> On Tue, Mar 14, 2017 at 1:57 PM, Josh Elser<elserj@apache.org>
>>>>>  wrote:
>>>>>
>>>>> I took a moment to read through the "blockers" as originally
>>>>> identified by
>>>>>
>>>>>> Vlad, and (to echo Enis' take) I read the majority of them as being
>>>>>> blockers not for the next release, but for a "full-fledged feature".
>>>>>> I'm
>>>>>> going to intentional avoid addressing the discussion of shipping
>>>>>> partial
>>>>>> features (related, but not relevant at the moment).
>>>>>>
>>>>>> HBASE-15227 is actually the one that bothers me the most, with
>>>>>> HBASE-17133
>>>>>> coming in close behind. Vlad, is there any documentation you can
>>>>>> point me
>>>>>> to about what the current issues are with the current implementation?
>>>>>> For
>>>>>> example, what happens now if the system has some kind of "partial
>>>>>> completion state"?
>>>>>>
>>>>>> Documentation is one of those that is really hard to judge. We want to
>>>>>> get
>>>>>> this code out for people to use (and to free up our strained dev
>>>>>> resources), but what good is some feature if the docs are
>>>>>> missing/incomplete?
>>>>>>
>>>>>> I think I could stomach the docs being inaccurate (with a clear
>>>>>> disclaimer
>>>>>> that the chapter is incomplete -- that's a 5min task). But, I think I
>>>>>> need
>>>>>> an answer about how the feature handles our common dist-sys category
>>>>>> of
>>>>>> problems before I can consider whether I'm ok with the feature hitting
>>>>>> 2.0...
>>>>>>
>>>>>> I'll also try to throw up a few nodes and play with it to address the
>>>>>> problem as an (ignorant) user ;)
>>>>>>
>>>>>>
>>>>>> Andrew Purtell wrote:
>>>>>>
>>>>>> I don't like that issues were identified as "blockers" but now there
>>>>>> is
>>>>>>
>>>>>>> an
>>>>>>> attempt to walk that back.
>>>>>>>
>>>>>>> I don't like that development of this feature has lingered for a long
>>>>>>> time
>>>>>>> in this unfinished state when this work could have been done by now,
>>>>>>> now
>>>>>>> that we are trying to get a 2.0 out the door. Because this is a
>>>>>>> volunteer
>>>>>>> project I cannot make any demand that it should be done, but I can
>>>>>>> certainly look at the current state and be nonplussed. This will be
>>>>>>> yet
>>>>>>> another half finished thing in 2.0 when this merge happens. Promises
>>>>>>> to
>>>>>>> finish the unfinished work are nice but not currency. Commits are
>>>>>>> currency.
>>>>>>> I hope at least the fault tolerance changes can be completed and
>>>>>>> committed
>>>>>>> before we spin a 2.0 RC, and without causing a 2.0 release to slip
>>>>>>> further.
>>>>>>>
>>>>>>> Also, marking something experimental should be done on the merits of
>>>>>>> that
>>>>>>> evaluation, not simply to justify dropping unfinished work into a
>>>>>>> release
>>>>>>> branch.
>>>>>>>
>>>>>>> I will change my vote to -0.
>>>>>>>
>>>>>>>
>>>>>>> On Mon, Mar 13, 2017 at 4:05 PM, Enis Söztutar<enis@apache.org>
>>>>>>>   wrote:
>>>>>>>
>>>>>>> I think there is some misconception of using the term "blockers" for
>>>>>>>
>>>>>>> referring to those jiras. My understanding is that those three jiras
>>>>>>>> are
>>>>>>>> blockers for the backup functionality to be more mature and more
>>>>>>>> usable.
>>>>>>>> But they are not release blockers. Let's say we merged the code in,
>>>>>>>> and
>>>>>>>> for
>>>>>>>> some reason those did not get addressed in time. We can still do the
>>>>>>>> 2.0
>>>>>>>> release without having to wait for the commits. We can instead mark
>>>>>>>> the
>>>>>>>> "backup" feature as experimental with known issues and go on with
>>>>>>>> the
>>>>>>>> release. In that sense they are not real release blockers.
>>>>>>>>
>>>>>>>> We are proposing the merge at this time because of the above that
>>>>>>>> maintaining this in a branch is becoming extremely costly and not
>>>>>>>> productive for the HBase community. Realistically, we cannot have
>>>>>>>> the
>>>>>>>> luxury of having to wait another couple of months and doing yet
>>>>>>>> another
>>>>>>>> giant round of reviews because the code base is a moving target.
>>>>>>>>
>>>>>>>> Enis
>>>>>>>>
>>>>>>>> On Mon, Mar 13, 2017 at 3:46 PM, Devaraj Das<ddas@hortonworks.com>
>>>>>>>> wrote:
>>>>>>>>
>>>>>>>> Vlad, on the first point, I think what Stack is saying is that
>>>>>>>> creating
>>>>>>>>
>>>>>>>> the new branch (as Ted did) ignores the feedback incorporated thus
>>>>>>>>> far
>>>>>>>>> in
>>>>>>>>> the iterations of the mega-patch. That's a wrong way to go.
>>>>>>>>> On the separation into a backup module, again, that was reverted to
>>>>>>>>> ease
>>>>>>>>> reviews of the mega-patch, and was noted as work to be done later.
>>>>>>>>> I
>>>>>>>>>
>>>>>>>>> think
>>>>>>>>>
>>>>>>>> Stack just wants to make the list of remaining work more complete by
>>>>>>>>
>>>>>>>>> citing
>>>>>>>>>
>>>>>>>> that as pending work.
>>>>>>>>
>>>>>>>>> ________________________________________
>>>>>>>>> From: Vladimir Rodionov<vladrodionov@gmail.com>
>>>>>>>>> Sent: Monday, March 13, 2017 3:09 PM
>>>>>>>>> To: dev@hbase.apache.org
>>>>>>>>> Subject: Re: [VOTE] Backup/Restore feature for HBase 2.0, vote
>>>>>>>>> closing
>>>>>>>>> 3/11/2017
>>>>>>>>>
>>>>>>>>>     It ignores the feedback
>>>>>>>>>
>>>>>>>>> If I "ignore" feedback, I put my comment - why? I am always  open
>>>>>>>>>> for
>>>>>>>>>>
>>>>>>>>>> further discussions. If reviewer does not insist on a particular
>>>>>>>>> request
>>>>>>>>>
>>>>>>>>> -
>>>>>>>>>
>>>>>>>> it will be dropped. I think it is fair.
>>>>>>>>
>>>>>>>>> he list is incomplete because a bunch of
>>>>>>>>>
>>>>>>>>> follow-ons came of the review cycle (including moving
>>>>>>>>>> backup/restore
>>>>>>>>>>
>>>>>>>>>>> out
>>>>>>>>>>>
>>>>>>>>>> of
>>>>>>>>>
>>>>>>>>> core to live in its own module).
>>>>>>>>>
>>>>>>>>>> For those who were not following our lengthy conversation on a
>>>>>>>>>> review
>>>>>>>>>>
>>>>>>>>>> board, separation of a backup code into a separate module has been
>>>>>>>>> done last year, but has been reverted back by request of a
>>>>>>>>> reviewer.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> -Vladimir
>>>>>>>>>
>>>>>>>>> On Mon, Mar 13, 2017 at 2:23 PM, Stack<stack@duboce.net>    wrote:
>>>>>>>>>
>>>>>>>>> On Fri, Mar 10, 2017 at 9:09 PM, Stack<stack@duboce.net>    wrote:
>>>>>>>>>
>>>>>>>>> On Fri, Mar 10, 2017 at 6:01 PM, Ted Yu<yuzhihong@gmail.com>
>>>>>>>>>>   wrote:
>>>>>>>>>>
>>>>>>>>>> HBASE-14123 branch has been created, with Vlad's mega patch v61.
>>>>>>>>>>>
>>>>>>>>>>> The patch put up for VOTE here was done on a branch. The call to
>>>>>>>>>>>>
>>>>>>>>>>>> merge
>>>>>>>>>>>
>>>>>>>>>> seems to have been premature given the many cycles of review and
>>>>>>>>> test
>>>>>>>>>
>>>>>>>>> that
>>>>>>>>>>
>>>>>>>>>> happened subsequent (The cycles burned a bunch of dev resource).
>>>>>>>>>>
>>>>>>>>>>> The patch as is is now in a state where it is too big for our
>>>>>>>>>>> infra;
>>>>>>>>>>>
>>>>>>>>>>> rb
>>>>>>>>>>>
>>>>>>>>>> and JIRA are creaking under the size and # of iterations.
>>>>>>>>>
>>>>>>>>> Adding finish of new JIRAs to this merge implies a new round of
>>>>>>>>>>
>>>>>>>>>>> review
>>>>>>>>>>>
>>>>>>>>>> and
>>>>>>>>>
>>>>>>>>> test of an already massive patch. Who is going to do this work?
>>>>>>>>>>
>>>>>>>>>>> Going back to a new branch seems wrong route to take.
>>>>>>>>>>>
>>>>>>>>>>> St.Ack
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> To be more explicit, this patch was developed on a branch and
>>>>>>>>>>> then a
>>>>>>>>>>>
>>>>>>>>>>> bunch
>>>>>>>>>>
>>>>>>>>> of dev resources were burned getting it into a state where it
>>>>>>>>> could be
>>>>>>>>>
>>>>>>>>>> merged to master. Going back to a branch to bulk up the merge so
>>>>>>>>>> it
>>>>>>>>>> includes more JIRAs than the many it already incorporates is the
>>>>>>>>>> wrong
>>>>>>>>>> direction for us to be headed in. It ignores the feedback given
>>>>>>>>>> and
>>>>>>>>>> the
>>>>>>>>>> work done by Vladimir slimming down an already over-broad scope.
>>>>>>>>>> It
>>>>>>>>>> is
>>>>>>>>>>
>>>>>>>>>> also
>>>>>>>>>>
>>>>>>>>> predicated on abundant review and testing resource being on tap to
>>>>>>>>>
>>>>>>>>>> cycle
>>>>>>>>>>
>>>>>>>>> on
>>>>>>>>>
>>>>>>>>> a feature that is useful, but non-core.
>>>>>>>>>
>>>>>>>>>> The patch is ready for merge IMO. Geoffrey makes a nice list of
>>>>>>>>>> what
>>>>>>>>>> is
>>>>>>>>>> still to do though IIRC, the list is incomplete because a bunch of
>>>>>>>>>> follow-ons came of the review cycle (including moving
>>>>>>>>>> backup/restore
>>>>>>>>>>
>>>>>>>>>> out
>>>>>>>>>>
>>>>>>>>> of
>>>>>>>>>
>>>>>>>>> core to live in its own module).
>>>>>>>>>
>>>>>>>>>> The patch needs three votes to merge. I am not against merge but
>>>>>>>>>> I am
>>>>>>>>>>
>>>>>>>>>> not
>>>>>>>>>>
>>>>>>>>> voting for the patch because I do have any more time to spend on
>>>>>>>>> this
>>>>>>>>>
>>>>>>>>> non-core feature and feel that a vote will have me assume a
>>>>>>>>>>
>>>>>>>>>> responsibility
>>>>>>>>>>
>>>>>>>>> I will not shirk.
>>>>>>>>>
>>>>>>>>>> S
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> FYI
>>>>>>>>>>
>>>>>>>>>>> On Fri, Mar 10, 2017 at 3:30 PM, Ted Yu<yuzhihong@gmail.com>
>>>>>>>>>>>>
>>>>>>>>>>>> wrote:
>>>>>>>>>>>>
>>>>>>>>>>> Thanks for the feedback, Andrew.
>>>>>>>>>> How about the following plan:
>>>>>>>>>>
>>>>>>>>>>> create branch HBASE-14123 off of master with mega patch v61 as
>>>>>>>>>>>>> the
>>>>>>>>>>>>>
>>>>>>>>>>>>> first
>>>>>>>>>>>>>
>>>>>>>>>>>> commit (reviewed by Stack and Enis)
>>>>>>>>>>>
>>>>>>>>>>> Vlad and I continue development (the 3 blockers) based on
>>>>>>>>>>>>
>>>>>>>>>>>>> HBASE-14123
>>>>>>>>>>>>>
>>>>>>>>>>>>     branch
>>>>>>>>>>> when all of the blockers get +1 and merged into HBASE-14123
>>>>>>>>>>>
>>>>>>>>>>>> branch,
>>>>>>>>>>>>>
>>>>>>>>>>>> we
>>>>>>>>>>>
>>>>>>>>>> propose to community for merging into branch-2 (master branch, if
>>>>>>>>>
>>>>>>>>>> branch-2
>>>>>>>>>>>
>>>>>>>>>>>> doesn't materialize for whatever reason) again
>>>>>>>>>>>>
>>>>>>>>>>>>> Cheers
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> On Fri, Mar 10, 2017 at 3:01 PM, Andrew Purtell<
>>>>>>>>>>>>>
>>>>>>>>>>>>> apurtell@apache.org>
>>>>>>>>>>>>>
>>>>>>>>>>>> wrote:
>>>>>>>>>>> Thanks for the offer but I like that you were honest about
>>>>>>>>>>>
>>>>>>>>>>>> compiling
>>>>>>>>>>>>>
>>>>>>>>>>>>> a
>>>>>>>>>>>>
>>>>>>>>>>> list
>>>>>>>>>>
>>>>>>>>>>> of issues that you thought were blockers for release. Since this
>>>>>>>>>>>>
>>>>>>>>>>>>> proposal
>>>>>>>>>>>>>>
>>>>>>>>>>>>> is a merge into 2.0, and we are trying to release 2.0, I am -1
>>>>>>>>>>>>> on
>>>>>>>>>>>>> this
>>>>>>>>>>>>>
>>>>>>>>>>>>> merge until those blockers are addressed.
>>>>>>>>>>>> I had a look at the list.
>>>>>>>>>>>>
>>>>>>>>>>>>> I think the documentation issue is important but not actually a
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> blocker.
>>>>>>>>>>>>>>
>>>>>>>>>>>>> That may be a controversial opinion, but documentation can be
>>>>>>>>>>>>> back-filled
>>>>>>>>>>>>> worst case. So take HBASE-17133 off the list.
>>>>>>>>>>>>>
>>>>>>>>>>>>> Remaining are effectively HBASE-14417, HBASE-14141, and
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> HBASE-15227.
>>>>>>>>>>>>>>
>>>>>>>>>>>>> They
>>>>>>>>>>>>
>>>>>>>>>>> all have patches attached to the respective JIRAs so completing
>>>>>>>>>>>
>>>>>>>>>>>> this
>>>>>>>>>>>>>
>>>>>>>>>>>>> work
>>>>>>>>>>>>
>>>>>>>>>>> won't be onerous. Get these committed and I will lift my -1. The
>>>>>>>>>>>
>>>>>>>>>>>> others
>>>>>>>>>>>>>
>>>>>>>>>>>>> who
>>>>>>>>>>>> voted +1 on this thread surely can help with that.
>>>>>>>>>>>>
>>>>>>>>>>>>> Thanks.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> On Fri, Mar 10, 2017 at 2:32 PM, Vladimir Rodionov<
>>>>>>>>>>>>>> vladrodionov@gmail.com>
>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> No problem I will downgrade Blockers to Majors if it scares
>>>>>>>>>>>>>> you,
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Andrew
>>>>>>>>>>>>>
>>>>>>>>>>>> [image: [image: 🙂]]
>>>>>>>>>>
>>>>>>>>>> Sent from my iPhone
>>>>>>>>>>>>>
>>>>>>>>>>>>>> On Mar 10, 2017, at 1:52 PM, Andrew Purtell<
>>>>>>>>>>>>>>> apurtell@apache.org
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>
>>>>>>>>>>>>> ​I know the merge of this feature has lagged substantially. I
>>>>>>>>>>>
>>>>>>>>>>>> think
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> that
>>>>>>>>>>>>>>
>>>>>>>>>>>>> is
>>>>>>>>>>>>
>>>>>>>>>>>>> regrettable but on another thread we are lamenting that 2.0
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> is
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> already
>>>>>>>>>>>>>>
>>>>>>>>>>>>> late. Unless I misunderstand, this is a proposal to merge
>>>>>>>>>>
>>>>>>>>>>> something
>>>>>>>>>>>>>> with
>>>>>>>>>>>>>>
>>>>>>>>>>>>> known blockers into trunk before we branch it for 2.0 which
>>>>>>>>>>>>
>>>>>>>>>>>>> will
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> effectively prevent that release because these blockers will
>>>>>>>>>>>>>>
>>>>>>>>>>>>> be
>>>>>>>>>>>
>>>>>>>>>>>> there. I
>>>>>>>>>>>>>>
>>>>>>>>>>>>> am
>>>>>>>>>>
>>>>>>>>>>> inclined to veto. Probably we should not propose branch
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> merges
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> into
>>>>>>>>>>>>>>
>>>>>>>>>>>>> code
>>>>>>>>>>
>>>>>>>>>>> we
>>>>>>>>>>>>
>>>>>>>>>>>>> are trying to get out the door with known blockers. Why not
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> do
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> that
>>>>>>>>>>>>>>
>>>>>>>>>>>>> work
>>>>>>>>>>
>>>>>>>>>>> first? It seems an obvious question. Perhaps I am missing
>>>>>>>>>>>>
>>>>>>>>>>>>> something.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> If we can branch for 2.0 now and then merge this, and not
>>>>>>>>>>>>>> into
>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>
>>>>>>>>>>>>> 2.0
>>>>>>>>>>
>>>>>>>>>>> branch, I would vote +1 for branch merge even with known
>>>>>>>>>>>>
>>>>>>>>>>>>> blockers
>>>>>>>>>>>>>> pending.
>>>>>>>>>>>>>>
>>>>>>>>>>>>> ​
>>>>>>>>>>>
>>>>>>>>>>>> On Fri, Mar 10, 2017 at 1:42 PM, Vladimir Rodionov<
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> vladrodionov@gmail.com>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> They are not blockers for merge - only for 2.0. GA
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> As I said already the feature is usable right now
>>>>>>>>>>>>>>>>> We would like to continue working on master and we would
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> like
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> to
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>> see
>>>>>>>>>
>>>>>>>>>> a
>>>>>>>>>>>
>>>>>>>>>>>> commitment from community
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Sent from my iPhone
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> On Mar 10, 2017, at 11:16 AM, Andrew Purtell<
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> apurtell@apache.org
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Only BLOCKERs and CRITICALs are guaranteed for HBase 2.0
>>>>>>>>>>>>
>>>>>>>>>>>>> release.
>>>>>>>>>>>>>>>>> If we have identified blockers, why merge this before they
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> are
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> in?
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Otherwise we can't release 2.0, and it is overdue.
>>>>>>>>>>>
>>>>>>>>>>>> On Wed, Mar 8, 2017 at 1:32 PM, Vladimir Rodionov<
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> vladrodionov@gmail.com>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Hello, HBase folks
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> For your consideration today is Backup/Restore feature for
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> Apache
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> HBAse
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> 2.0.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Backup code is available as a mega patch in HBASE-14123
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> (v61),
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> applies
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> cleanly to the current master, all test PASS, patch has no
>>>>>>>>>>>
>>>>>>>>>>>> other
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> issues.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> The patch has gone through numerous rounds of code reviews
>>>>>>>>>>>>
>>>>>>>>>>>>> and
>>>>>>>>>>>>>>>>> has
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> probably
>>>>>>>>>>>
>>>>>>>>>>>> the most lengthy discussion thread on Apache JIRA
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> (HBASE-14123)
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> :)
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> The work has been split into 3 phases (HBASE-14030, 14123,
>>>>>>>>>>>>
>>>>>>>>>>>>> 14414)
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Two
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> first
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> are complete, third one is still in progress.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> *** Summary of work HBASE-14123
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> The new feature introduces new command-line extensions to
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> hbase
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> command
>>>>>>>>>>>
>>>>>>>>>>>> and, from the client side, is accessible through
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> command-line
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> only
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Operations:
>>>>>>>>>>>
>>>>>>>>>>>> * Create full backup on a list of tables or backup set
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> * Create incremental backup image for table list or backup
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> set
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> * Restore list of tables from a given backup image
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> * Show current backup progress
>>>>>>>>>>>
>>>>>>>>>>>> * Delete backup image and all related images
>>>>>>>>>>>>>>>>>>> * Show history of backups
>>>>>>>>>>>>>>>>>>> * Backup set operations: create backup set, add/remove
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> table
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> to/from
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> backup
>>>>>>>>>>
>>>>>>>>>>> set, etc
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> In the current implementation, the feature is already
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> usable,
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> meaning
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> that
>>>>>>>>>>>
>>>>>>>>>>>> users can backup tables and restore them using provided
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> command-line
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> tools.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Both: full and incremental backups are supported.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> This work is based on original work of IBM team
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> (HBASE-7912).
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> The
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> full
>>>>>>>>>>>
>>>>>>>>>>>> list
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> of JIRAs included in this mega patch can be found in three
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> umbrella
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> JIRAs:
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> HBASE-14030 (Phase 1), HBASE-14123 (Phase 2) and
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> HBASE-14414
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> (Phase 3
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> -
>>>>>>>>>>
>>>>>>>>>>> all
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> resolved ones made it into the patch)
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> *** What are the remaining work items
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> All remaining items can be found in Phase 3 umbrella
>>>>>>>>>>>>>>>>>>> JIRA:
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> HBASE-14414.
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> They are split into 3 groups: BLOCKER, CRITICAL, MAJOR
>>>>>>>>>>>>>>>>> Only BLOCKERs and CRITICALs are guaranteed for HBase 2.0
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> release.
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> ***** BLOCKER
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> * HBASE-14417 Incremental backup and bulk loading ( Patch
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> available)
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> * HBASE-14135 HBase Backup/Restore Phase 3: Merge backup
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> images
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> * HBASE-14141 HBase Backup/Restore Phase 3: Filter WALs on
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> backup
>>>>>>>>>>>>
>>>>>>>>>>>>> to
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> include only edits from backup tables (Patch available)
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> * HBASE-17133 Backup documentation
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> * HBASE-15227 Fault tolerance support
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> ***** CRITICAL
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> * HBASE-16465 Disable split/merges during backup
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> We have umbrella JIRA (HBASE-14414) to track all the
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> remaining
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> work
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> All the BLOCKER and CRITICAL JIRAs currently in open state
>>>>>>>>>>>
>>>>>>>>>>>> will
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> be
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> implemented by 2.0 release time. Some MAJOR too, but it
>>>>>>>>>>>>
>>>>>>>>>>>>> depends
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> on
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> resource
>>>>>>>>>>>>
>>>>>>>>>>>>> availability
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> The former development branch (HBASE-7912) is obsolete and
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> will
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> be
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> closed/deleted after the merge.
>>>>>>>>>>>>
>>>>>>>>>>>>> We want backup to be a GA feature in 2.0
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> We are going to support full backward compatibility for
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> backup
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> tool in
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> 2.0
>>>>>>>>>>>
>>>>>>>>>>>> and onwards.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> **** Configuration
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> Backup is disabled, by default. To enable it, the
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> following
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> configuration
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> properties must be added to hbase-site.xml:
>>>>>>>>>>
>>>>>>>>>>> hbase.backup.enable=true
>>>>>>>>>>>>>>>>>>> hbase.master.logcleaner.plugins=YOUR_PLUGINS,org.
>>>>>>>>>>>>>>>>>>> apache.hadoop.hbase.backup.master.BackupLogCleaner
>>>>>>>>>>>>>>>>>>> hbase.procedure.master.classes=YOUR_CLASSES,org.
>>>>>>>>>>>>>>>>>>> apache.hadoop.hbase.backup.master.
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> LogRollMasterProcedureManager
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> hbase.procedure.regionserver.classes=YOUR_CLASSES,org.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> apache.hadoop.hbase.backup.regionserver.
>>>>>>>>>>>>
>>>>>>>>>>>>> LogRollRegionServerProcedureMa
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> nager
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> I would like to thank IBM team and Jerry He for original
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> work,
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Enis, Ted, Stack, Matteo, Jerry for time spent on code
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> reviews
>>>>>>>>>>>
>>>>>>>>>>>> Special thanks to Ted Yu for his co-development work.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> References:
>>>>>>>>>>>
>>>>>>>>>>>> https://issues.apache.org/jira/browse/HBASE-7912
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> (original
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> IBM,
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> contains
>>>>>>>>>>
>>>>>>>>>>> design doc)
>>>>>>>>>>>>
>>>>>>>>>>>>> https://issues.apache.org/jira/browse/HBASE-14030 (Phase
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> 1)
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> https://issues.apache.org/jira/browse/HBASE-14123 (Phase
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> 2)
>>>>>>>>>>
>>>>>>>>>>> https://issues.apache.org/jira/browse/HBASE-14414 (Phase
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> 3)
>>>>>>>>>>
>>>>>>>>>>> Please  vote +1/-1 by midnight Pacific Time (00:00
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> -0800 GMT) on March 11th  ​on whether or not we should
>>>>>>>>>>
>>>>>>>>>>> merge
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> this
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> into
>>>>>>>>>>
>>>>>>>>>>> the
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> current master.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> -Vladimir Rodionov
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> --
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Best regards,
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>     - Andy
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> If you are given a choice, you believe you have acted
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> freely. -
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Raymond
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Teller (via Peter Watts)
>>>>>>>>>>>
>>>>>>>>>>>> --
>>>>>>>>>>>>>>>> Best regards,
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>      - Andy
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> If you are given a choice, you believe you have acted
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> freely. -
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Raymond
>>>>>>>>>>>>>>
>>>>>>>>>>>>> Teller (via Peter Watts)
>>>>>>>>>>
>>>>>>>>>>> --
>>>>>>>>>>>>>> Best regards,
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>       - Andy
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> If you are given a choice, you believe you have acted freely.
>>>>>>>>>>>>>> -
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Raymond
>>>>>>>>>>>>>>
>>>>>>>>>>>>> Teller (via Peter Watts)
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>
>>>>>
>>>
>>>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message