hbase-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Josh Elser <els...@apache.org>
Subject Re: [DISCUSSION] Merge Backup / Restore - Branch HBASE-7912
Date Fri, 08 Sep 2017 22:23:30 GMT
Based on the list of stuff on HBASE-14414 and offline-chats had with 
Vlad myself, I know of the following being needed

https://issues.apache.org/jira/browse/HBASE-15227 - Fault tolerance 
umbrella (no-op on its own)
https://issues.apache.org/jira/browse/HBASE-17852 - I believe Vlad is 
working on this one now
https://issues.apache.org/jira/browse/HBASE-16465 - disable 
splits/merges (marked as required for 15227 -- not sure if that's 
accurate anymore)
https://issues.apache.org/jira/browse/HBASE-17851 -- Just a unit test
https://issues.apache.org/jira/browse/HBASE-17133 - Docs

Echo'ing Stack, we're at the point where we need to draw a line in the 
sand for what will hit 2.0.0 to avoid mucking up testing.

Have I grasped the state of things correctly, Vlad?

On 9/8/17 5:39 PM, Stack wrote:
> HBASE-14414 is a JIRA with a list of random seeming issues w/ non-descript
> summaries: "Add nonce support to TableBackupProcedure, BackupID must
> include backup set name, ...". The last comment in that issue is from July.
> It asks: "How do I figure what of backup/restore feature is going to be in
> hbase-2.0.0? Thanks Vladimir Rodionov
> <https://issues.apache.org/jira/secure/ViewProfile.jspa?name=vrodionov>."
> to which there is no answer.  Doc update is TODO.
> 
> Where is the summary of the capability in hbase-2? What testing and at what
> scale has testing been done? Is this 'stable or experimental'? If I can't
> get basic info on this feature though I ask repeatedly, what hope does the
> poor old operator have?
> 
> St.Ack
> 
> 
> On Fri, Sep 8, 2017 at 1:59 PM, Vladimir Rodionov <vladrodionov@gmail.com>
> wrote:
> 
>> HBASE-14414
>>
>> On Fri, Sep 8, 2017 at 1:14 PM, Stack <stack@duboce.net> wrote:
>>
>>> Where do I go to get the current status of this feature? Looking in JIRA
>> I
>>> see loads of issues open against backup including some against
>> hbase-2.0.0
>>> and no progress being made that I can discern.
>>>
>>> Thanks,
>>> S
>>>
>>>
>>>
>>> On Wed, Nov 23, 2016 at 8:52 AM, Stack <stack@duboce.net> wrote:
>>>
>>>> On Tue, Nov 22, 2016 at 6:48 PM, Stack <stack@duboce.net> wrote:
>>>>
>>>>> On Tue, Nov 22, 2016 at 3:17 PM, Vladimir Rodionov <
>>>>> vladrodionov@gmail.com> wrote:
>>>>>
>>>>>>>> and/or he answered most of the review feedback
>>>>>>
>>>>>> No, questions are still open, but I do not see any blockers and we
>> have
>>>>>> HBASE-16940 to address these questions.
>>>>>>
>>>>>>
>>>>> Agree. No blockers but stuff that should be dealt with (No one will
>> pay
>>>>> me any attention once merge goes in -- smile).
>>>>>
>>>>>
>>>> Let me clarify the above. I want review addressed before merge happens.
>>>> Sorry if any confusion.
>>>> St.Ack
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>> St.Ack
>>>>>
>>>>>
>>>>>
>>>>>> On Tue, Nov 22, 2016 at 3:04 PM, Devaraj Das <ddas@hortonworks.com>
>>>>>> wrote:
>>>>>>
>>>>>>> Hi Stack, hats off to you for spending so much time on this!
>> Thanks!
>>>>>> From
>>>>>>> my understanding, Vlad has raised follow-up jiras for the issues
>> you
>>>>>>> raised, and/or he answered most of the review feedback. So, do
you
>>>>>> think we
>>>>>>> could do a merge vote now?
>>>>>>> Devaraj.
>>>>>>> ________________________________________
>>>>>>> From: Vladimir Rodionov <vladrodionov@gmail.com>
>>>>>>> Sent: Monday, November 21, 2016 8:34 PM
>>>>>>> To: dev@hbase.apache.org
>>>>>>> Subject: Re: [DISCUSSION] Merge Backup / Restore - Branch
>> HBASE-7912
>>>>>>>
>>>>>>>>> I have spent a good bit of time reviewing and testing
this
>>> feature.
>>>>>> I
>>>>>>> would
>>>>>>>>> like my review and concerns addressed and I'd like it
to be
>> clear
>>>>>> how;
>>>>>>>>> either explicit follow-on issues, pointers to where in
the patch
>>> or
>>>>>> doc
>>>>>>> my
>>>>>>>>> remarks have been catered to, etc. Until then, I am against
>>> commit.
>>>>>>>
>>>>>>> Stack, mega patch review comments will be addressed in the
>> dedicated
>>>>>> JIRA:
>>>>>>> HBASE-16940
>>>>>>> I have open several other JIRAs to address your other comments
(not
>>> on
>>>>>>> review board).
>>>>>>>
>>>>>>> Details are here (end of the thread):
>>>>>>> https://issues.apache.org/jira/browse/HBASE-14123
>>>>>>>
>>>>>>> Let me know what else should we do to move merge forward.
>>>>>>>
>>>>>>> -Vlad
>>>>>>>
>>>>>>>
>>>>>>> On Fri, Nov 18, 2016 at 4:54 PM, Stack <stack@duboce.net>
wrote:
>>>>>>>
>>>>>>>> On Fri, Nov 18, 2016 at 3:53 PM, Ted Yu <yuzhihong@gmail.com>
>>> wrote:
>>>>>>>>
>>>>>>>>> Thanks, Matteo.
>>>>>>>>>
>>>>>>>>> bq. restore is not clear if given an incremental id it
will do
>>> the
>>>>>> full
>>>>>>>>> restore from full up to that point or if i need to apply
>> manually
>>>>>>>>> everything
>>>>>>>>>
>>>>>>>>> The restore takes into consideration of the dependent
>> backup(s).
>>>>>>>>> So there is no need to apply preceding backup(s) manually.
>>>>>>>>>
>>>>>>>>>
>>>>>>>> I ask this question on the issue. It is not clear from the
usage
>> or
>>>>>> doc
>>>>>>> how
>>>>>>>> to run a restore from incremental. Can you fix in doc and
usage
>> how
>>>>>> so I
>>>>>>>> can be clear and try it. Currently I am stuck verifying a
round
>>> trip
>>>>>>> backup
>>>>>>>> restore made of incrementals.
>>>>>>>>
>>>>>>>> Thanks,
>>>>>>>> S
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>> On Fri, Nov 18, 2016 at 3:48 PM, Matteo Bertozzi <
>>>>>>>> theo.bertozzi@gmail.com>
>>>>>>>>> wrote:
>>>>>>>>>
>>>>>>>>>> I did one last pass to the mega patch. I don't see
anything
>>> major
>>>>>>> that
>>>>>>>>>> should block the merge.
>>>>>>>>>>
>>>>>>>>>> - most of the code is isolated in the backup package
>>>>>>>>>> - all the backup code is client side
>>>>>>>>>> - there are few changes to the server side, mainly
for
>>> cleaners,
>>>>>> wal
>>>>>>>>>> rolling and similar (which is ok)
>>>>>>>>>> - there is a good number of tests, and an integration
test
>>>>>>>>>>
>>>>>>>>>> the code seems to have still some left overs from
the old
>>>>>>>> implementation,
>>>>>>>>>> and some stuff needs a cleanup. but I don't think
this should
>>> be
>>>>>> used
>>>>>>>> as
>>>>>>>>> an
>>>>>>>>>> argument to block the merge. I think the guys will
keep
>> working
>>>>>> on
>>>>>>> this
>>>>>>>>> and
>>>>>>>>>> they may also get help of others once the patch is
in master.
>>>>>>>>>>
>>>>>>>>>> I still have my concerns about the current limitations,
but
>>>>>> these are
>>>>>>>>>> things already planned for phase 3, so some of this
stuff may
>>>>>> even be
>>>>>>>> in
>>>>>>>>>> the final 2.0.
>>>>>>>>>> but as long as we have a "current limitations" section
in the
>>>>>> user
>>>>>>>> guide
>>>>>>>>>> mentioning important stuff like the ones below, I'm
ok with
>> it.
>>>>>>>>>>   - if you write to the table with Durability.SKIP_WALS
your
>>> data
>>>>>> will
>>>>>>>> not
>>>>>>>>>> be in the incremental-backup
>>>>>>>>>>   - if you bulkload files that data will not be in
the
>>> incremental
>>>>>>>> backup
>>>>>>>>>> (HBASE-14417)
>>>>>>>>>>   - the incremental backup will not only contains
the data of
>>> the
>>>>>>> table
>>>>>>>>> you
>>>>>>>>>> specified but also the regions from other tables
that are on
>>> the
>>>>>> same
>>>>>>>> set
>>>>>>>>>> of RSs (HBASE-14141) ...maybe a note about security
around
>> this
>>>>>> topic
>>>>>>>>>>   - the incremental backup will not contains just
the "latest
>>> row"
>>>>>>>> between
>>>>>>>>>> backup A and B, but it will also contains all the
updates
>>>>>> occurred in
>>>>>>>>>> between. but the restore does not allow you to restore
up to
>> a
>>>>>>> certain
>>>>>>>>>> point in time, the restore will always be up to the
"latest
>>>>>> backup
>>>>>>>>> point".
>>>>>>>>>>   - you should limit the number of "incremental"
up to N (or
>>> maybe
>>>>>>>> SIZE),
>>>>>>>>> to
>>>>>>>>>> avoid replay time becoming the bottleneck. (HBASE-14135)
>>>>>>>>>>
>>>>>>>>>> I'll be ok even with the above not being in the final
2.0,
>>>>>>>>>> but i'd like to see as blocker for the final 2.0
(not the
>>> merge)
>>>>>>>>>>   - the backup code moved in an hbase-backup module
>>>>>>>>>>   - and some more work around tools, especially to
try to
>> unify
>>>>>> and
>>>>>>> make
>>>>>>>>>> simple the backup experience (simple example: in
some case
>>> there
>>>>>> is a
>>>>>>>>>> backup_id argument in others a backupId argument.
or things
>>>>>> like..
>>>>>>>>> restore
>>>>>>>>>> is not clear if given an incremental id it will do
the full
>>>>>> restore
>>>>>>>> from
>>>>>>>>>> full up to that point or if i need to apply manually
>>> everything).
>>>>>>>>>>
>>>>>>>>>> in conclusion, I think we can open a merge vote.
I'll be +1
>> on
>>>>>> it,
>>>>>>> and
>>>>>>>> I
>>>>>>>>>> think we should try to reject -1 with just a "code
cleanup"
>>>>>>> motivation,
>>>>>>>>>> since there will still be work going on on the code
after the
>>>>>> merge.
>>>>>>>>>>
>>>>>>>>>> Matteo
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> On Sun, Nov 6, 2016 at 10:54 PM, Devaraj Das <
>>>>>> ddas@hortonworks.com>
>>>>>>>>> wrote:
>>>>>>>>>>
>>>>>>>>>>> Stack and others, anything else on the patch?
Merge to
>> master
>>>>>> now?
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>
>>>>>
>>>>>
>>>>
>>>
>>
> 

Mime
View raw message