Return-Path: X-Original-To: archive-asf-public-internal@cust-asf2.ponee.io Delivered-To: archive-asf-public-internal@cust-asf2.ponee.io Received: from cust-asf.ponee.io (cust-asf.ponee.io [163.172.22.183]) by cust-asf2.ponee.io (Postfix) with ESMTP id 66A10200C3D for ; Tue, 14 Mar 2017 21:57:14 +0100 (CET) Received: by cust-asf.ponee.io (Postfix) id 61DFD160B7E; Tue, 14 Mar 2017 20:57:14 +0000 (UTC) Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by cust-asf.ponee.io (Postfix) with SMTP id 368CB160B63 for ; Tue, 14 Mar 2017 21:57:13 +0100 (CET) Received: (qmail 85575 invoked by uid 500); 14 Mar 2017 20:57:12 -0000 Mailing-List: contact dev-help@hbase.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@hbase.apache.org Delivered-To: mailing list dev@hbase.apache.org Received: (qmail 85564 invoked by uid 99); 14 Mar 2017 20:57:11 -0000 Received: from mail-relay.apache.org (HELO mail-relay.apache.org) (140.211.11.15) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 14 Mar 2017 20:57:11 +0000 Received: from hw10447.local (pool-68-134-11-100.bltmmd.fios.verizon.net [68.134.11.100]) by mail-relay.apache.org (ASF Mail Server at mail-relay.apache.org) with ESMTPSA id 2D4111A0193 for ; Tue, 14 Mar 2017 20:57:10 +0000 (UTC) Message-ID: <58C85921.6060007@apache.org> Date: Tue, 14 Mar 2017 16:57:05 -0400 From: Josh Elser User-Agent: Postbox 3.0.11 (Macintosh/20140602) MIME-Version: 1.0 To: dev@hbase.apache.org Subject: Re: [VOTE] Backup/Restore feature for HBase 2.0, vote closing 3/11/2017 References: <9085FD13-AD7C-48F1-9ED1-D5F5F5101A38@gmail.com> <0B4C81CE-B579-4FC5-A015-0253792129A0@gmail.com> <1489445161992.55748@hortonworks.com> In-Reply-To: Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 8bit archived-at: Tue, 14 Mar 2017 20:57:14 -0000 I took a moment to read through the "blockers" as originally identified by Vlad, and (to echo Enis' take) I read the majority of them as being blockers not for the next release, but for a "full-fledged feature". I'm going to intentional avoid addressing the discussion of shipping partial features (related, but not relevant at the moment). HBASE-15227 is actually the one that bothers me the most, with HBASE-17133 coming in close behind. Vlad, is there any documentation you can point me to about what the current issues are with the current implementation? For example, what happens now if the system has some kind of "partial completion state"? Documentation is one of those that is really hard to judge. We want to get this code out for people to use (and to free up our strained dev resources), but what good is some feature if the docs are missing/incomplete? I think I could stomach the docs being inaccurate (with a clear disclaimer that the chapter is incomplete -- that's a 5min task). But, I think I need an answer about how the feature handles our common dist-sys category of problems before I can consider whether I'm ok with the feature hitting 2.0... I'll also try to throw up a few nodes and play with it to address the problem as an (ignorant) user ;) Andrew Purtell wrote: > I don't like that issues were identified as "blockers" but now there is an > attempt to walk that back. > > I don't like that development of this feature has lingered for a long time > in this unfinished state when this work could have been done by now, now > that we are trying to get a 2.0 out the door. Because this is a volunteer > project I cannot make any demand that it should be done, but I can > certainly look at the current state and be nonplussed. This will be yet > another half finished thing in 2.0 when this merge happens. Promises to > finish the unfinished work are nice but not currency. Commits are currency. > I hope at least the fault tolerance changes can be completed and committed > before we spin a 2.0 RC, and without causing a 2.0 release to slip further. > > Also, marking something experimental should be done on the merits of that > evaluation, not simply to justify dropping unfinished work into a release > branch. > > I will change my vote to -0. > > > On Mon, Mar 13, 2017 at 4:05 PM, Enis Söztutar wrote: > >> I think there is some misconception of using the term "blockers" for >> referring to those jiras. My understanding is that those three jiras are >> blockers for the backup functionality to be more mature and more usable. >> But they are not release blockers. Let's say we merged the code in, and for >> some reason those did not get addressed in time. We can still do the 2.0 >> release without having to wait for the commits. We can instead mark the >> "backup" feature as experimental with known issues and go on with the >> release. In that sense they are not real release blockers. >> >> We are proposing the merge at this time because of the above that >> maintaining this in a branch is becoming extremely costly and not >> productive for the HBase community. Realistically, we cannot have the >> luxury of having to wait another couple of months and doing yet another >> giant round of reviews because the code base is a moving target. >> >> Enis >> >> On Mon, Mar 13, 2017 at 3:46 PM, Devaraj Das wrote: >> >>> Vlad, on the first point, I think what Stack is saying is that creating >>> the new branch (as Ted did) ignores the feedback incorporated thus far in >>> the iterations of the mega-patch. That's a wrong way to go. >>> On the separation into a backup module, again, that was reverted to ease >>> reviews of the mega-patch, and was noted as work to be done later. I >> think >>> Stack just wants to make the list of remaining work more complete by >> citing >>> that as pending work. >>> ________________________________________ >>> From: Vladimir Rodionov >>> Sent: Monday, March 13, 2017 3:09 PM >>> To: dev@hbase.apache.org >>> Subject: Re: [VOTE] Backup/Restore feature for HBase 2.0, vote closing >>> 3/11/2017 >>> >>>>> It ignores the feedback >>> If I "ignore" feedback, I put my comment - why? I am always open for >>> further discussions. If reviewer does not insist on a particular request >> - >>> it will be dropped. I think it is fair. >>> >>>>> he list is incomplete because a bunch of >>>>> follow-ons came of the review cycle (including moving backup/restore >> out >>> of >>>>> core to live in its own module). >>> For those who were not following our lengthy conversation on a review >>> board, separation of a backup code into a separate module has been >>> done last year, but has been reverted back by request of a reviewer. >>> >>> >>> -Vladimir >>> >>> On Mon, Mar 13, 2017 at 2:23 PM, Stack wrote: >>> >>>> On Fri, Mar 10, 2017 at 9:09 PM, Stack wrote: >>>> >>>>> On Fri, Mar 10, 2017 at 6:01 PM, Ted Yu wrote: >>>>> >>>>>> HBASE-14123 branch has been created, with Vlad's mega patch v61. >>>>>> >>>>> The patch put up for VOTE here was done on a branch. The call to >> merge >>>>> seems to have been premature given the many cycles of review and test >>>> that >>>>> happened subsequent (The cycles burned a bunch of dev resource). >>>>> >>>>> The patch as is is now in a state where it is too big for our infra; >> rb >>>>> and JIRA are creaking under the size and # of iterations. >>>>> >>>>> Adding finish of new JIRAs to this merge implies a new round of >> review >>>> and >>>>> test of an already massive patch. Who is going to do this work? >>>>> >>>>> Going back to a new branch seems wrong route to take. >>>>> >>>>> St.Ack >>>>> >>>>> >>>>> >>>> To be more explicit, this patch was developed on a branch and then a >>> bunch >>>> of dev resources were burned getting it into a state where it could be >>>> merged to master. Going back to a branch to bulk up the merge so it >>>> includes more JIRAs than the many it already incorporates is the wrong >>>> direction for us to be headed in. It ignores the feedback given and the >>>> work done by Vladimir slimming down an already over-broad scope. It is >>> also >>>> predicated on abundant review and testing resource being on tap to >> cycle >>> on >>>> a feature that is useful, but non-core. >>>> >>>> The patch is ready for merge IMO. Geoffrey makes a nice list of what is >>>> still to do though IIRC, the list is incomplete because a bunch of >>>> follow-ons came of the review cycle (including moving backup/restore >> out >>> of >>>> core to live in its own module). >>>> >>>> The patch needs three votes to merge. I am not against merge but I am >> not >>>> voting for the patch because I do have any more time to spend on this >>>> non-core feature and feel that a vote will have me assume a >>> responsibility >>>> I will not shirk. >>>> >>>> S >>>> >>>> >>>> >>>>> >>>>> >>>>>> FYI >>>>>> >>>>>> On Fri, Mar 10, 2017 at 3:30 PM, Ted Yu >> wrote: >>>>>>> Thanks for the feedback, Andrew. >>>>>>> >>>>>>> How about the following plan: >>>>>>> >>>>>>> create branch HBASE-14123 off of master with mega patch v61 as the >>>> first >>>>>>> commit (reviewed by Stack and Enis) >>>>>>> Vlad and I continue development (the 3 blockers) based on >>> HBASE-14123 >>>>>>> branch >>>>>>> when all of the blockers get +1 and merged into HBASE-14123 >> branch, >>> we >>>>>>> propose to community for merging into branch-2 (master branch, if >>>>>> branch-2 >>>>>>> doesn't materialize for whatever reason) again >>>>>>> >>>>>>> Cheers >>>>>>> >>>>>>> >>>>>>> On Fri, Mar 10, 2017 at 3:01 PM, Andrew Purtell< >>> apurtell@apache.org> >>>>>>> wrote: >>>>>>> >>>>>>>> Thanks for the offer but I like that you were honest about >>> compiling >>>> a >>>>>>>> list >>>>>>>> of issues that you thought were blockers for release. Since this >>>>>> proposal >>>>>>>> is a merge into 2.0, and we are trying to release 2.0, I am -1 on >>>> this >>>>>>>> merge until those blockers are addressed. >>>>>>>> >>>>>>>> I had a look at the list. >>>>>>>> >>>>>>>> I think the documentation issue is important but not actually a >>>>>> blocker. >>>>>>>> That may be a controversial opinion, but documentation can be >>>>>> back-filled >>>>>>>> worst case. So take HBASE-17133 off the list. >>>>>>>> >>>>>>>> Remaining are effectively HBASE-14417, HBASE-14141, and >>> HBASE-15227. >>>>>> They >>>>>>>> all have patches attached to the respective JIRAs so completing >>> this >>>>>> work >>>>>>>> won't be onerous. Get these committed and I will lift my -1. The >>>> others >>>>>>>> who >>>>>>>> voted +1 on this thread surely can help with that. >>>>>>>> >>>>>>>> Thanks. >>>>>>>> >>>>>>>> >>>>>>>> On Fri, Mar 10, 2017 at 2:32 PM, Vladimir Rodionov< >>>>>>>> vladrodionov@gmail.com> >>>>>>>> wrote: >>>>>>>> >>>>>>>>> No problem I will downgrade Blockers to Majors if it scares >> you, >>>>>> Andrew >>>>>>>> 🙂 >>>>>>>>> Sent from my iPhone >>>>>>>>> >>>>>>>>>> On Mar 10, 2017, at 1:52 PM, Andrew Purtell< >>> apurtell@apache.org >>>>>>>> wrote: >>>>>>>>>> ​I know the merge of this feature has lagged substantially. I >>>> think >>>>>>>> that >>>>>>>>> is >>>>>>>>>> regrettable but on another thread we are lamenting that 2.0 >> is >>>>>> already >>>>>>>>>> late. Unless I misunderstand, this is a proposal to merge >>>> something >>>>>>>> with >>>>>>>>>> known blockers into trunk before we branch it for 2.0 which >>> will >>>>>>>>>> effectively prevent that release because these blockers will >> be >>>>>>>> there. I >>>>>>>>> am >>>>>>>>>> inclined to veto. Probably we should not propose branch >> merges >>>> into >>>>>>>> code >>>>>>>>> we >>>>>>>>>> are trying to get out the door with known blockers. Why not >> do >>>> that >>>>>>>> work >>>>>>>>>> first? It seems an obvious question. Perhaps I am missing >>>>>> something. >>>>>>>>>> If we can branch for 2.0 now and then merge this, and not >> into >>>> the >>>>>> 2.0 >>>>>>>>>> branch, I would vote +1 for branch merge even with known >>> blockers >>>>>>>>> pending. >>>>>>>>>> ​ >>>>>>>>>> >>>>>>>>>> On Fri, Mar 10, 2017 at 1:42 PM, Vladimir Rodionov< >>>>>>>>> vladrodionov@gmail.com> >>>>>>>>>> wrote: >>>>>>>>>> >>>>>>>>>>> They are not blockers for merge - only for 2.0. GA >>>>>>>>>>> As I said already the feature is usable right now >>>>>>>>>>> We would like to continue working on master and we would >> like >>> to >>>>>> see >>>>>>>> a >>>>>>>>>>> commitment from community >>>>>>>>>>> >>>>>>>>>>> Sent from my iPhone >>>>>>>>>>> >>>>>>>>>>> On Mar 10, 2017, at 11:16 AM, Andrew Purtell< >>>> apurtell@apache.org >>>>>>>>> wrote: >>>>>>>>>>>>> Only BLOCKERs and CRITICALs are guaranteed for HBase 2.0 >>>>>> release. >>>>>>>>>>>> If we have identified blockers, why merge this before they >>> are >>>>>> in? >>>>>>>>>>>> Otherwise we can't release 2.0, and it is overdue. >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> On Wed, Mar 8, 2017 at 1:32 PM, Vladimir Rodionov< >>>>>>>>>>> vladrodionov@gmail.com> >>>>>>>>>>>> wrote: >>>>>>>>>>>> >>>>>>>>>>>>> Hello, HBase folks >>>>>>>>>>>>> >>>>>>>>>>>>> For your consideration today is Backup/Restore feature for >>>>>> Apache >>>>>>>>> HBAse >>>>>>>>>>>>> 2.0. >>>>>>>>>>>>> Backup code is available as a mega patch in HBASE-14123 >>> (v61), >>>>>>>> applies >>>>>>>>>>>>> cleanly to the current master, all test PASS, patch has no >>>> other >>>>>>>>> issues. >>>>>>>>>>>>> The patch has gone through numerous rounds of code reviews >>> and >>>>>> has >>>>>>>>>>> probably >>>>>>>>>>>>> the most lengthy discussion thread on Apache JIRA >>>> (HBASE-14123) >>>>>> :) >>>>>>>>>>>>> The work has been split into 3 phases (HBASE-14030, 14123, >>>>>> 14414) >>>>>>>> Two >>>>>>>>>>> first >>>>>>>>>>>>> are complete, third one is still in progress. >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> *** Summary of work HBASE-14123 >>>>>>>>>>>>> >>>>>>>>>>>>> The new feature introduces new command-line extensions to >>> the >>>>>> hbase >>>>>>>>>>> command >>>>>>>>>>>>> and, from the client side, is accessible through >>> command-line >>>>>> only >>>>>>>>>>>>> Operations: >>>>>>>>>>>>> * Create full backup on a list of tables or backup set >>>>>>>>>>>>> * Create incremental backup image for table list or backup >>> set >>>>>>>>>>>>> * Restore list of tables from a given backup image >>>>>>>>>>>>> * Show current backup progress >>>>>>>>>>>>> * Delete backup image and all related images >>>>>>>>>>>>> * Show history of backups >>>>>>>>>>>>> * Backup set operations: create backup set, add/remove >> table >>>>>>>> to/from >>>>>>>>>>> backup >>>>>>>>>>>>> set, etc >>>>>>>>>>>>> >>>>>>>>>>>>> In the current implementation, the feature is already >>> usable, >>>>>>>> meaning >>>>>>>>>>> that >>>>>>>>>>>>> users can backup tables and restore them using provided >>>>>>>> command-line >>>>>>>>>>> tools. >>>>>>>>>>>>> Both: full and incremental backups are supported. >>>>>>>>>>>>> This work is based on original work of IBM team >>> (HBASE-7912). >>>>>> The >>>>>>>> full >>>>>>>>>>> list >>>>>>>>>>>>> of JIRAs included in this mega patch can be found in three >>>>>> umbrella >>>>>>>>>>> JIRAs: >>>>>>>>>>>>> HBASE-14030 (Phase 1), HBASE-14123 (Phase 2) and >> HBASE-14414 >>>>>>>> (Phase 3 >>>>>>>>> - >>>>>>>>>>> all >>>>>>>>>>>>> resolved ones made it into the patch) >>>>>>>>>>>>> >>>>>>>>>>>>> *** What are the remaining work items >>>>>>>>>>>>> >>>>>>>>>>>>> All remaining items can be found in Phase 3 umbrella JIRA: >>>>>>>>> HBASE-14414. >>>>>>>>>>>>> They are split into 3 groups: BLOCKER, CRITICAL, MAJOR >>>>>>>>>>>>> Only BLOCKERs and CRITICALs are guaranteed for HBase 2.0 >>>>>> release. >>>>>>>>>>>>> ***** BLOCKER >>>>>>>>>>>>> >>>>>>>>>>>>> * HBASE-14417 Incremental backup and bulk loading ( Patch >>>>>>>> available) >>>>>>>>>>>>> * HBASE-14135 HBase Backup/Restore Phase 3: Merge backup >>>> images >>>>>>>>>>>>> * HBASE-14141 HBase Backup/Restore Phase 3: Filter WALs on >>>>>> backup >>>>>>>> to >>>>>>>>>>>>> include only edits from backup tables (Patch available) >>>>>>>>>>>>> * HBASE-17133 Backup documentation >>>>>>>>>>>>> * HBASE-15227 Fault tolerance support >>>>>>>>>>>>> >>>>>>>>>>>>> ***** CRITICAL >>>>>>>>>>>>> >>>>>>>>>>>>> * HBASE-16465 Disable split/merges during backup >>>>>>>>>>>>> >>>>>>>>>>>>> We have umbrella JIRA (HBASE-14414) to track all the >>> remaining >>>>>> work >>>>>>>>>>>>> All the BLOCKER and CRITICAL JIRAs currently in open state >>>> will >>>>>> be >>>>>>>>>>>>> implemented by 2.0 release time. Some MAJOR too, but it >>>> depends >>>>>> on >>>>>>>>>>> resource >>>>>>>>>>>>> availability >>>>>>>>>>>>> The former development branch (HBASE-7912) is obsolete and >>>> will >>>>>> be >>>>>>>>>>>>> closed/deleted after the merge. >>>>>>>>>>>>> We want backup to be a GA feature in 2.0 >>>>>>>>>>>>> We are going to support full backward compatibility for >>> backup >>>>>>>> tool in >>>>>>>>>>> 2.0 >>>>>>>>>>>>> and onwards. >>>>>>>>>>>>> >>>>>>>>>>>>> **** Configuration >>>>>>>>>>>>> >>>>>>>>>>>>> Backup is disabled, by default. To enable it, the >> following >>>>>>>>>>> configuration >>>>>>>>>>>>> properties must be added to hbase-site.xml: >>>>>>>>>>>>> >>>>>>>>>>>>> hbase.backup.enable=true >>>>>>>>>>>>> hbase.master.logcleaner.plugins=YOUR_PLUGINS,org. >>>>>>>>>>>>> apache.hadoop.hbase.backup.master.BackupLogCleaner >>>>>>>>>>>>> hbase.procedure.master.classes=YOUR_CLASSES,org. >>>>>>>>>>>>> apache.hadoop.hbase.backup.master. >>>> LogRollMasterProcedureManager >>>>>>>>>>>>> hbase.procedure.regionserver.classes=YOUR_CLASSES,org. >>>>>>>>>>>>> apache.hadoop.hbase.backup.regionserver. >>>>>>>>> LogRollRegionServerProcedureMa >>>>>>>>>>>>> nager >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> I would like to thank IBM team and Jerry He for original >>> work, >>>>>>>>>>>>> Enis, Ted, Stack, Matteo, Jerry for time spent on code >>> reviews >>>>>>>>>>>>> Special thanks to Ted Yu for his co-development work. >>>>>>>>>>>>> >>>>>>>>>>>>> References: >>>>>>>>>>>>> >>>>>>>>>>>>> https://issues.apache.org/jira/browse/HBASE-7912 >> (original >>>> IBM, >>>>>>>>>>> contains >>>>>>>>>>>>> design doc) >>>>>>>>>>>>> https://issues.apache.org/jira/browse/HBASE-14030 (Phase >> 1) >>>>>>>>>>>>> https://issues.apache.org/jira/browse/HBASE-14123 (Phase >> 2) >>>>>>>>>>>>> https://issues.apache.org/jira/browse/HBASE-14414 (Phase >> 3) >>>>>>>>>>>>> Please vote +1/-1 by midnight Pacific Time (00:00 >>>>>>>>>>>>> -0800 GMT) on March 11th ​on whether or not we should >> merge >>>>>> this >>>>>>>> into >>>>>>>>>>> the >>>>>>>>>>>>> current master. >>>>>>>>>>>>> >>>>>>>>>>>>> -Vladimir Rodionov >>>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> -- >>>>>>>>>>>> Best regards, >>>>>>>>>>>> >>>>>>>>>>>> - Andy >>>>>>>>>>>> >>>>>>>>>>>> If you are given a choice, you believe you have acted >>> freely. - >>>>>>>> Raymond >>>>>>>>>>>> Teller (via Peter Watts) >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> -- >>>>>>>>>> Best regards, >>>>>>>>>> >>>>>>>>>> - Andy >>>>>>>>>> >>>>>>>>>> If you are given a choice, you believe you have acted >> freely. - >>>>>>>> Raymond >>>>>>>>>> Teller (via Peter Watts) >>>>>>>> >>>>>>>> >>>>>>>> -- >>>>>>>> Best regards, >>>>>>>> >>>>>>>> - Andy >>>>>>>> >>>>>>>> If you are given a choice, you believe you have acted freely. - >>>> Raymond >>>>>>>> Teller (via Peter Watts) >>>>>>>> >>>>>>> >>>>> > > >