hbase-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Andrew Purtell <andrew.purt...@gmail.com>
Subject Re: [DISCUSSION] Merge Backup / Restore - Branch HBASE-7912
Date Sat, 09 Sep 2017 23:40:28 GMT
For what it's worth, I think AMv2 is the main reason to have a 2.0 in the first place, so I
would both agree it needs a lot more testing and yet I would want us to have a 2.0 release
as the vehicle for getting that to happen. For other features without testing from a number
of parties or at scale the value proposition is less clear and it's fine by me for the RM
to set them aside for future releases. 

Also, I can relay that there is some interest where I work in utilizing HBASE-7912 but the
impression we have is it is unfinished and untested. So for now we are ignoring it and continuing
with home grown solutions. Part of the problem is fault tolerance was left to the last phase(s)
and yet it is an essential property for adoption for serious work. The best way to resolve
this IMHO is for the developers of this feature to complete those unfinished JIRAs, especially
concerning resilience to failures. 


> On Sep 9, 2017, at 4:11 PM, Vladimir Rodionov <vladrodionov@gmail.com> wrote:
> 
> Hmm, the next on your list (of kicked out from branch v2) should be AM v2 I
> presume?
> 
> -Vlad
> 
>> On Sat, Sep 9, 2017 at 4:04 PM, stack <saint.ack@gmail.com> wrote:
>> 
>> In spite of repeated requests for eng summary of state of this feature --
>> summary of what is in 2.0, what is not, what the capabilities are, how well
>> it has been tested and at what scale -- all I get, when the requests are
>> not ignored, are pointers to lists of ill-describing jiras and some pending
>> user facing doc update.
>> 
>> For other features, mob or region server groups, I know that they have been
>> running at scale in production for as much as a year and more. I have some
>> confidence these items basically work.  For backup/restore I have no such
>> sense even after spending time in review and trying to use the feature.
>> 
>> As release manager, I have say over what makes it into a release.  Unless
>> the work is done to convince me that backup/restore is more than a lump of
>> code and a few unit tests that can pass on some fellows laptop, I am going
>> to kick it out of branch-2.  Let the feature harden more in master branch
>> before it ships in a release.
>> 
>> S
>> 
>> On Sep 8, 2017 10:59 PM, "Vladimir Rodionov" <vladrodionov@gmail.com>
>> wrote:
>> 
>>>>> Have I grasped the state of things correctly, Vlad?
>>> 
>>> Josh, the only thing which is still pending is doc update. All other
>>> features are good to have but not a blockers for 2.0 release.
>>> 
>>> -Vlad
>>> 
>>> On Fri, Sep 8, 2017 at 10:42 PM, Vladimir Rodionov <
>> vladrodionov@gmail.com
>>>> 
>>> wrote:
>>> 
>>>>>> What testing and at what
>>>>>> scale has testing been done?
>>>> 
>>>> Do we have have that for other features?
>>>> 
>>>> 
>>>> On Fri, Sep 8, 2017 at 10:41 PM, Vladimir Rodionov <
>>> vladrodionov@gmail.com
>>>>> wrote:
>>>> 
>>>>>>> It asks: "How do I figure what of backup/restore feature is going
>> to
>>>>> be in
>>>>>>> hbase-2.0.0?
>>>>> 
>>>>> Hmm, wait for doc update.
>>>>> 
>>>>> 
>>>>>> On Fri, Sep 8, 2017 at 2:39 PM, Stack <stack@duboce.net> wrote:
>>>>>> 
>>>>>> HBASE-14414 is a JIRA with a list of random seeming issues w/
>>>>>> non-descript
>>>>>> summaries: "Add nonce support to TableBackupProcedure, BackupID must
>>>>>> include backup set name, ...". The last comment in that issue is
from
>>>>>> July.
>>>>>> It asks: "How do I figure what of backup/restore feature is going
to
>> be
>>>>>> in
>>>>>> hbase-2.0.0? Thanks Vladimir Rodionov
>>>>>> <https://issues.apache.org/jira/secure/ViewProfile.jspa?
>> name=vrodionov
>>>>>>> ."
>>>>>> to which there is no answer.  Doc update is TODO.
>>>>>> 
>>>>>> Where is the summary of the capability in hbase-2? What testing and
>> at
>>>>>> what
>>>>>> scale has testing been done? Is this 'stable or experimental'? If
I
>>> can't
>>>>>> get basic info on this feature though I ask repeatedly, what hope
>> does
>>>>>> the
>>>>>> poor old operator have?
>>>>>> 
>>>>>> St.Ack
>>>>>> 
>>>>>> 
>>>>>> On Fri, Sep 8, 2017 at 1:59 PM, Vladimir Rodionov <
>>>>>> vladrodionov@gmail.com>
>>>>>> wrote:
>>>>>> 
>>>>>>> HBASE-14414
>>>>>>> 
>>>>>>>> On Fri, Sep 8, 2017 at 1:14 PM, Stack <stack@duboce.net>
wrote:
>>>>>>>> 
>>>>>>>> Where do I go to get the current status of this feature?
Looking
>> in
>>>>>> JIRA
>>>>>>> I
>>>>>>>> see loads of issues open against backup including some against
>>>>>>> hbase-2.0.0
>>>>>>>> and no progress being made that I can discern.
>>>>>>>> 
>>>>>>>> Thanks,
>>>>>>>> S
>>>>>>>> 
>>>>>>>> 
>>>>>>>> 
>>>>>>>>> On Wed, Nov 23, 2016 at 8:52 AM, Stack <stack@duboce.net>
wrote:
>>>>>>>>> 
>>>>>>>>> On Tue, Nov 22, 2016 at 6:48 PM, Stack <stack@duboce.net>
>> wrote:
>>>>>>>>> 
>>>>>>>>>> On Tue, Nov 22, 2016 at 3:17 PM, Vladimir Rodionov
<
>>>>>>>>>> vladrodionov@gmail.com> wrote:
>>>>>>>>>> 
>>>>>>>>>>>>> and/or he answered most of the review
feedback
>>>>>>>>>>> 
>>>>>>>>>>> No, questions are still open, but I do not see
any blockers
>> and
>>>>>> we
>>>>>>> have
>>>>>>>>>>> HBASE-16940 to address these questions.
>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>> Agree. No blockers but stuff that should be dealt
with (No one
>>>>>> will
>>>>>>> pay
>>>>>>>>>> me any attention once merge goes in -- smile).
>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>> Let me clarify the above. I want review addressed before
merge
>>>>>> happens.
>>>>>>>>> Sorry if any confusion.
>>>>>>>>> St.Ack
>>>>>>>>> 
>>>>>>>>> 
>>>>>>>>> 
>>>>>>>>> 
>>>>>>>>> 
>>>>>>>>> 
>>>>>>>>>> St.Ack
>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>>>> On Tue, Nov 22, 2016 at 3:04 PM, Devaraj Das
<
>>>>>> ddas@hortonworks.com>
>>>>>>>>>>> wrote:
>>>>>>>>>>> 
>>>>>>>>>>>> Hi Stack, hats off to you for spending so
much time on
>> this!
>>>>>>> Thanks!
>>>>>>>>>>> From
>>>>>>>>>>>> my understanding, Vlad has raised follow-up
jiras for the
>>>>>> issues
>>>>>>> you
>>>>>>>>>>>> raised, and/or he answered most of the review
feedback. So,
>>> do
>>>>>> you
>>>>>>>>>>> think we
>>>>>>>>>>>> could do a merge vote now?
>>>>>>>>>>>> Devaraj.
>>>>>>>>>>>> ________________________________________
>>>>>>>>>>>> From: Vladimir Rodionov <vladrodionov@gmail.com>
>>>>>>>>>>>> Sent: Monday, November 21, 2016 8:34 PM
>>>>>>>>>>>> To: dev@hbase.apache.org
>>>>>>>>>>>> Subject: Re: [DISCUSSION] Merge Backup /
Restore - Branch
>>>>>>> HBASE-7912
>>>>>>>>>>>> 
>>>>>>>>>>>>>> I have spent a good bit of time reviewing
and testing
>> this
>>>>>>>> feature.
>>>>>>>>>>> I
>>>>>>>>>>>> would
>>>>>>>>>>>>>> like my review and concerns addressed
and I'd like it to
>>> be
>>>>>>> clear
>>>>>>>>>>> how;
>>>>>>>>>>>>>> either explicit follow-on issues,
pointers to where in
>> the
>>>>>> patch
>>>>>>>> or
>>>>>>>>>>> doc
>>>>>>>>>>>> my
>>>>>>>>>>>>>> remarks have been catered to, etc.
Until then, I am
>>> against
>>>>>>>> commit.
>>>>>>>>>>>> 
>>>>>>>>>>>> Stack, mega patch review comments will be
addressed in the
>>>>>>> dedicated
>>>>>>>>>>> JIRA:
>>>>>>>>>>>> HBASE-16940
>>>>>>>>>>>> I have open several other JIRAs to address
your other
>>> comments
>>>>>> (not
>>>>>>>> on
>>>>>>>>>>>> review board).
>>>>>>>>>>>> 
>>>>>>>>>>>> Details are here (end of the thread):
>>>>>>>>>>>> https://issues.apache.org/jira/browse/HBASE-14123
>>>>>>>>>>>> 
>>>>>>>>>>>> Let me know what else should we do to move
merge forward.
>>>>>>>>>>>> 
>>>>>>>>>>>> -Vlad
>>>>>>>>>>>> 
>>>>>>>>>>>> 
>>>>>>>>>>>> On Fri, Nov 18, 2016 at 4:54 PM, Stack <stack@duboce.net>
>>>>>> wrote:
>>>>>>>>>>>> 
>>>>>>>>>>>>> On Fri, Nov 18, 2016 at 3:53 PM, Ted
Yu <
>>> yuzhihong@gmail.com
>>>>>>> 
>>>>>>>> wrote:
>>>>>>>>>>>>> 
>>>>>>>>>>>>>> Thanks, Matteo.
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> bq. restore is not clear if given
an incremental id it
>>>>>> will do
>>>>>>>> the
>>>>>>>>>>> full
>>>>>>>>>>>>>> restore from full up to that point
or if i need to
>> apply
>>>>>>> manually
>>>>>>>>>>>>>> everything
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> The restore takes into consideration
of the dependent
>>>>>>> backup(s).
>>>>>>>>>>>>>> So there is no need to apply preceding
backup(s)
>>> manually.
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> 
>>>>>>>>>>>>> I ask this question on the issue. It
is not clear from
>> the
>>>>>> usage
>>>>>>> or
>>>>>>>>>>> doc
>>>>>>>>>>>> how
>>>>>>>>>>>>> to run a restore from incremental. Can
you fix in doc and
>>>>>> usage
>>>>>>> how
>>>>>>>>>>> so I
>>>>>>>>>>>>> can be clear and try it. Currently I
am stuck verifying a
>>>>>> round
>>>>>>>> trip
>>>>>>>>>>>> backup
>>>>>>>>>>>>> restore made of incrementals.
>>>>>>>>>>>>> 
>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>> S
>>>>>>>>>>>>> 
>>>>>>>>>>>>> 
>>>>>>>>>>>>> 
>>>>>>>>>>>>>> On Fri, Nov 18, 2016 at 3:48 PM,
Matteo Bertozzi <
>>>>>>>>>>>>> theo.bertozzi@gmail.com>
>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> I did one last pass to the mega
patch. I don't see
>>>>>> anything
>>>>>>>> major
>>>>>>>>>>>> that
>>>>>>>>>>>>>>> should block the merge.
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> - most of the code is isolated
in the backup package
>>>>>>>>>>>>>>> - all the backup code is client
side
>>>>>>>>>>>>>>> - there are few changes to the
server side, mainly
>> for
>>>>>>>> cleaners,
>>>>>>>>>>> wal
>>>>>>>>>>>>>>> rolling and similar (which is
ok)
>>>>>>>>>>>>>>> - there is a good number of tests,
and an integration
>>>>>> test
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> the code seems to have still
some left overs from the
>>> old
>>>>>>>>>>>>> implementation,
>>>>>>>>>>>>>>> and some stuff needs a cleanup.
but I don't think
>> this
>>>>>> should
>>>>>>>> be
>>>>>>>>>>> used
>>>>>>>>>>>>> as
>>>>>>>>>>>>>> an
>>>>>>>>>>>>>>> argument to block the merge.
I think the guys will
>> keep
>>>>>>> working
>>>>>>>>>>> on
>>>>>>>>>>>> this
>>>>>>>>>>>>>> and
>>>>>>>>>>>>>>> they may also get help of others
once the patch is in
>>>>>> master.
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> I still have my concerns about
the current
>> limitations,
>>>>>> but
>>>>>>>>>>> these are
>>>>>>>>>>>>>>> things already planned for phase
3, so some of this
>>>>>> stuff may
>>>>>>>>>>> even be
>>>>>>>>>>>>> in
>>>>>>>>>>>>>>> the final 2.0.
>>>>>>>>>>>>>>> but as long as we have a "current
limitations"
>> section
>>>>>> in the
>>>>>>>>>>> user
>>>>>>>>>>>>> guide
>>>>>>>>>>>>>>> mentioning important stuff like
the ones below, I'm
>> ok
>>>>>> with
>>>>>>> it.
>>>>>>>>>>>>>>> - if you write to the table with
>> Durability.SKIP_WALS
>>>>>> your
>>>>>>>> data
>>>>>>>>>>> will
>>>>>>>>>>>>> not
>>>>>>>>>>>>>>> be in the incremental-backup
>>>>>>>>>>>>>>> - if you bulkload files that
data will not be in the
>>>>>>>> incremental
>>>>>>>>>>>>> backup
>>>>>>>>>>>>>>> (HBASE-14417)
>>>>>>>>>>>>>>> - the incremental backup will
not only contains the
>>>>>> data of
>>>>>>>> the
>>>>>>>>>>>> table
>>>>>>>>>>>>>> you
>>>>>>>>>>>>>>> specified but also the regions
from other tables that
>>>>>> are on
>>>>>>>> the
>>>>>>>>>>> same
>>>>>>>>>>>>> set
>>>>>>>>>>>>>>> of RSs (HBASE-14141) ...maybe
a note about security
>>>>>> around
>>>>>>> this
>>>>>>>>>>> topic
>>>>>>>>>>>>>>> - the incremental backup will
not contains just the
>>>>>> "latest
>>>>>>>> row"
>>>>>>>>>>>>> between
>>>>>>>>>>>>>>> backup A and B, but it will also
contains all the
>>> updates
>>>>>>>>>>> occurred in
>>>>>>>>>>>>>>> between. but the restore does
not allow you to
>> restore
>>>>>> up to
>>>>>>> a
>>>>>>>>>>>> certain
>>>>>>>>>>>>>>> point in time, the restore will
always be up to the
>>>>>> "latest
>>>>>>>>>>> backup
>>>>>>>>>>>>>> point".
>>>>>>>>>>>>>>> - you should limit the number
of "incremental" up
>> to N
>>>>>> (or
>>>>>>>> maybe
>>>>>>>>>>>>> SIZE),
>>>>>>>>>>>>>> to
>>>>>>>>>>>>>>> avoid replay time becoming the
bottleneck.
>>> (HBASE-14135)
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> I'll be ok even with the above
not being in the final
>>>>>> 2.0,
>>>>>>>>>>>>>>> but i'd like to see as blocker
for the final 2.0 (not
>>> the
>>>>>>>> merge)
>>>>>>>>>>>>>>> - the backup code moved in an
hbase-backup module
>>>>>>>>>>>>>>> - and some more work around tools,
especially to try
>>> to
>>>>>>> unify
>>>>>>>>>>> and
>>>>>>>>>>>> make
>>>>>>>>>>>>>>> simple the backup experience
(simple example: in some
>>>>>> case
>>>>>>>> there
>>>>>>>>>>> is a
>>>>>>>>>>>>>>> backup_id argument in others
a backupId argument. or
>>>>>> things
>>>>>>>>>>> like..
>>>>>>>>>>>>>> restore
>>>>>>>>>>>>>>> is not clear if given an incremental
id it will do
>> the
>>>>>> full
>>>>>>>>>>> restore
>>>>>>>>>>>>> from
>>>>>>>>>>>>>>> full up to that point or if i
need to apply manually
>>>>>>>> everything).
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> in conclusion, I think we can
open a merge vote. I'll
>>> be
>>>>>> +1
>>>>>>> on
>>>>>>>>>>> it,
>>>>>>>>>>>> and
>>>>>>>>>>>>> I
>>>>>>>>>>>>>>> think we should try to reject
-1 with just a "code
>>>>>> cleanup"
>>>>>>>>>>>> motivation,
>>>>>>>>>>>>>>> since there will still be work
going on on the code
>>>>>> after the
>>>>>>>>>>> merge.
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> Matteo
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> On Sun, Nov 6, 2016 at 10:54
PM, Devaraj Das <
>>>>>>>>>>> ddas@hortonworks.com>
>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> Stack and others, anything
else on the patch? Merge
>>> to
>>>>>>> master
>>>>>>>>>>> now?
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> 
>>>>>>>>>>>>> 
>>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>> 
>>>>>>>> 
>>>>>>> 
>>>>>> 
>>>>> 
>>>>> 
>>>> 
>>> 
>> 

Mime
View raw message