hbase-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Vladimir Rodionov <vladrodio...@gmail.com>
Subject Re: [DISCUSSION] Merge Backup / Restore - Branch HBASE-7912
Date Wed, 20 Jul 2016 00:45:40 GMT
Created:
https://issues.apache.org/jira/browse/HBASE-16255

-Vlad

On Tue, Jul 19, 2016 at 5:41 PM, Vladimir Rodionov <vladrodionov@gmail.com>
wrote:

> Matteo, answers are below:
>
>
>>
>> I don't know if there are already docs, but that is another thing we may
>> want to get in with the merge.
>>
>
>
>> a minimal coverage at least on how to use the feature, and maybe calling
>> it
>> out as experimental?
>>
>>
> We have the doc, need to get OK to release it to Apache.
>
>
>
>> my main concern were around incremental backups.
>> I'm still not convinced around the fact that because the WALs contain
>> regions of multiple tables
>> the incremental backup will keep around WALs with some data that we don't
>> really want in the backup (for space or maybe security reason).
>>
>>
> We have a JIRA: HBASE-14141 (Phase 3)
>
>
>> then there was the question about for how long should I take incrementals,
>> before deciding that a fresh full backup is less costly in terms of space.
>> but I think this incremental merge/compaction was a feature on the roadmap
>> as Phase3.
>> which I think is ok to get later on,
>> maybe just call out a lifecycle example on the docs under "best
>> practices".
>>
>>
> Yes, Merge feature is in Phase 3 - HBASE-14135
>
>
>
>>
>> has anyone interested in using backups looked at the doc in HBASE-7912?
>> is the current design of incremental backup acceptable for everyone
>> wanting
>> to use this feature?
>> (maybe this should be a question for the @user list and not dev)
>>
>> is there anyone already using this feature or it is just dev testing it?
>> to me will be interesting having a use-case/workflow example,
>> to see if in the real world my concerns about incremental are not showing
>> up.
>>
>>
> No customers yet, the feature is still in development, but what we have
> right now is
> pretty much useable, at least, as technical preview feature.
>
>
> And yes, I am going to add IT.
>
>
> -Vlad
>
>
>
>> On Tue, Jul 19, 2016 at 1:35 PM, Ted Yu <yuzhihong@gmail.com> wrote:
>>
>> > Gentle ping on this subject.
>> >
>> > The changes are mostly non-intrusive.
>> >
>> > More comments are welcome.
>> >
>> > On Mon, Jul 11, 2016 at 9:29 PM, Vladimir Rodionov <
>> vladrodionov@gmail.com
>> > >
>> > wrote:
>> >
>> > > Not that hard, Andrew. I will open JIRA.
>> > >
>> > > -Vlad
>> > >
>> > > On Mon, Jul 11, 2016 at 8:46 PM, Andrew Purtell <
>> > andrew.purtell@gmail.com>
>> > > wrote:
>> > >
>> > > > How hard would it be to convert what you've been using to test end
>> to
>> > end
>> > > > during dev into an IT?
>> > > >
>> > > >
>> > > > On Jul 11, 2016, at 5:31 PM, Vladimir Rodionov <
>> vladrodionov@gmail.com
>> > >
>> > > > wrote:
>> > > >
>> > > > >>> Is there an integration test in hbase-it yet? If not,
any tips
>> on a
>> > > > >>> semi-automateable way to take backups and restore them?
>> > > > >
>> > > > > We do not have yet, but we have a lot of unit tests. We provide
2
>> API
>> > > for
>> > > > > backup:
>> > > > >
>> > > > > 1. Admin.getBackupAdmin
>> > > > >
>> > > > > 2. Command - line via hbase command.
>> > > > >
>> > > > > Everything is straightforward.
>> > > > >
>> > > > > -Vlad
>> > > > >
>> > > > >
>> > > > >
>> > > > >
>> > > > >> On Mon, Jul 11, 2016 at 5:23 PM, Dima Spivak <
>> dspivak@cloudera.com>
>> > > > wrote:
>> > > > >>
>> > > > >> Is there an integration test in hbase-it yet? If not, any
tips
>> on a
>> > > > >> semi-automateable way to take backups and restore them?
>> > > > >>
>> > > > >> -Dima
>> > > > >>
>> > > > >> On Mon, Jul 11, 2016 at 6:42 PM, Vladimir Rodionov <
>> > > > vladrodionov@gmail.com
>> > > > >> wrote:
>> > > > >>
>> > > > >>> Sorry, wrong links:
>> > > > >>> These are the phases:
>> > > > >>>
>> > > > >>> Phase 1:
>> > > > >>> https://issues.apache.org/jira/browse/HBASE-
>> > > > >>> <https://issues.apache.org/jira/browse/HBASE-14030>14030
>> > > > >>> Phase 2:
>> > > > >>> https://issues.apache.org/jira/browse/HBASE-
>> > > > >>> <https://issues.apache.org/jira/browse/HBASE-14123>14123
>> > > > >>> Phase 3:
>> > > > >>> https://issues.apache.org/jira/browse/HBASE-
>> > > > >>> <https://issues.apache.org/jira/browse/HBASE-14414>14414
>> > > > >>>
>> > > > >>> -Vlad
>> > > > >>>
>> > > > >>> On Mon, Jul 11, 2016 at 4:41 PM, Vladimir Rodionov <
>> > > > >> vladrodionov@gmail.com
>> > > > >>> wrote:
>> > > > >>>
>> > > > >>>> These are the phases:
>> > > > >>>>
>> > > > >>>> Phase 1:
>> > > > >>>> https://issues.apache.org/jira/browse/HBASE-
>> > > > >>>> <https://issues.apache.org/jira/browse/HBASE-7912>14030
>> > > > >>>> Phase 2:
>> > > > >>>> https://issues.apache.org/jira/browse/HBASE-
>> > > > >>>> <https://issues.apache.org/jira/browse/HBASE-7912>14123
>> > > > >>>> Phase 3:
>> > > > >>>> https://issues.apache.org/jira/browse/HBASE-
>> > > > >>>> <https://issues.apache.org/jira/browse/HBASE-7912>14414
>> > > > >>>>
>> > > > >>>> -Vlad
>> > > > >>>>
>> > > > >>>>
>> > > > >>>> On Mon, Jul 11, 2016 at 12:21 PM, Enis Söztutar
<
>> enis@apache.org>
>> > > > >> wrote:
>> > > > >>>>
>> > > > >>>>> As you guys may already be familiar, Vladimir,
Ted, Jerry and
>> > > others
>> > > > >>> have
>> > > > >>>>> been developing the backup / restore functionality
in a
>> series of
>> > > > >> issues
>> > > > >>>>> committed in the separate branch HBASE-7912[1].
>> > > > >>>>>
>> > > > >>>>> Backup / Restore functionality is tracked as
a 4-phase
>> project,
>> > and
>> > > > >> the
>> > > > >>>>> first two phases are complete and useable. We
are now working
>> on
>> > > > >> Phase 3
>> > > > >>>>> items, which are mostly improvements. We think
that the
>> current
>> > > code
>> > > > >> in
>> > > > >>>>> the
>> > > > >>>>> branch containing all Phase 1 and Phase 2 items,
and some
>> Phase 3
>> > > > >> items
>> > > > >>> is
>> > > > >>>>> useable on it's own, and we do not have to wait
for all the
>> > > > subtickets
>> > > > >>> to
>> > > > >>>>> be finished to make it completely useable (as
follow up
>> tickets
>> > are
>> > > > >>> mostly
>> > > > >>>>> improvements or optimizations). The improvements
in the works
>> are
>> > > all
>> > > > >>>>> backwards compatible with the existing stuff.
Thus, we would
>> like
>> > > to
>> > > > >>>>> propose that the branch HBASE-7912 be merged
into master.  The
>> > > parent
>> > > > >>> jira
>> > > > >>>>> has a design doc that goes into details about
the
>> implementation
>> > > and
>> > > > >>>>> design
>> > > > >>>>> choices in case you are interested[2].
>> > > > >>>>>
>> > > > >>>>> Most of the changes are largely non-intrusive
and confined to
>> the
>> > > > >>>>> backup subsystem.
>> > > > >>>>> The unit tests have been passing on manual runs
and we
>> > > (hortonworks)
>> > > > >>> have
>> > > > >>>>> been running the integration tests as well as
some other
>> > > shell-based
>> > > > >>>>> system
>> > > > >>>>> tests on a forked version of the code. Most of
the work has
>> been
>> > > > >>> reviewed
>> > > > >>>>> by 1, 2 or 3 committers already (mostly Ted,
myself and
>> Jerry).
>> > > > >>>>>
>> > > > >>>>> What do you guys think? Is it time to call a
vote? Any
>> concerns
>> > or
>> > > > >>>>> feedback
>> > > > >>>>> appreciated.
>> > > > >>>>>
>> > > > >>>>> [1] https://issues.apache.org/jira/browse/HBASE-7912
>> > > > >>>>> [2]
>> > > > >>
>> > > >
>> > >
>> >
>> https://issues.apache.org/jira/secure/attachment/12816339/HBaseBackupAndRestore%20-0.91.pdf
>> > > > >>>>>
>> > > > >>>>> Enis
>> > > > >>
>> > > >
>> > >
>> >
>>
>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message