hbase-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Vladimir Rodionov <vladrodio...@gmail.com>
Subject Re: [DISCUSSION] Merge Backup / Restore - Branch HBASE-7912
Date Wed, 20 Jul 2016 00:41:37 GMT
Matteo, answers are below:


>
> I don't know if there are already docs, but that is another thing we may
> want to get in with the merge.
>


> a minimal coverage at least on how to use the feature, and maybe calling it
> out as experimental?
>
>
We have the doc, need to get OK to release it to Apache.



> my main concern were around incremental backups.
> I'm still not convinced around the fact that because the WALs contain
> regions of multiple tables
> the incremental backup will keep around WALs with some data that we don't
> really want in the backup (for space or maybe security reason).
>
>
We have a JIRA: HBASE-14141 (Phase 3)


> then there was the question about for how long should I take incrementals,
> before deciding that a fresh full backup is less costly in terms of space.
> but I think this incremental merge/compaction was a feature on the roadmap
> as Phase3.
> which I think is ok to get later on,
> maybe just call out a lifecycle example on the docs under "best practices".
>
>
Yes, Merge feature is in Phase 3 - HBASE-14135



>
> has anyone interested in using backups looked at the doc in HBASE-7912?
> is the current design of incremental backup acceptable for everyone wanting
> to use this feature?
> (maybe this should be a question for the @user list and not dev)
>
> is there anyone already using this feature or it is just dev testing it?
> to me will be interesting having a use-case/workflow example,
> to see if in the real world my concerns about incremental are not showing
> up.
>
>
No customers yet, the feature is still in development, but what we have
right now is
pretty much useable, at least, as technical preview feature.


And yes, I am going to add IT.


-Vlad



> On Tue, Jul 19, 2016 at 1:35 PM, Ted Yu <yuzhihong@gmail.com> wrote:
>
> > Gentle ping on this subject.
> >
> > The changes are mostly non-intrusive.
> >
> > More comments are welcome.
> >
> > On Mon, Jul 11, 2016 at 9:29 PM, Vladimir Rodionov <
> vladrodionov@gmail.com
> > >
> > wrote:
> >
> > > Not that hard, Andrew. I will open JIRA.
> > >
> > > -Vlad
> > >
> > > On Mon, Jul 11, 2016 at 8:46 PM, Andrew Purtell <
> > andrew.purtell@gmail.com>
> > > wrote:
> > >
> > > > How hard would it be to convert what you've been using to test end to
> > end
> > > > during dev into an IT?
> > > >
> > > >
> > > > On Jul 11, 2016, at 5:31 PM, Vladimir Rodionov <
> vladrodionov@gmail.com
> > >
> > > > wrote:
> > > >
> > > > >>> Is there an integration test in hbase-it yet? If not, any
tips
> on a
> > > > >>> semi-automateable way to take backups and restore them?
> > > > >
> > > > > We do not have yet, but we have a lot of unit tests. We provide 2
> API
> > > for
> > > > > backup:
> > > > >
> > > > > 1. Admin.getBackupAdmin
> > > > >
> > > > > 2. Command - line via hbase command.
> > > > >
> > > > > Everything is straightforward.
> > > > >
> > > > > -Vlad
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >> On Mon, Jul 11, 2016 at 5:23 PM, Dima Spivak <
> dspivak@cloudera.com>
> > > > wrote:
> > > > >>
> > > > >> Is there an integration test in hbase-it yet? If not, any tips
on
> a
> > > > >> semi-automateable way to take backups and restore them?
> > > > >>
> > > > >> -Dima
> > > > >>
> > > > >> On Mon, Jul 11, 2016 at 6:42 PM, Vladimir Rodionov <
> > > > vladrodionov@gmail.com
> > > > >> wrote:
> > > > >>
> > > > >>> Sorry, wrong links:
> > > > >>> These are the phases:
> > > > >>>
> > > > >>> Phase 1:
> > > > >>> https://issues.apache.org/jira/browse/HBASE-
> > > > >>> <https://issues.apache.org/jira/browse/HBASE-14030>14030
> > > > >>> Phase 2:
> > > > >>> https://issues.apache.org/jira/browse/HBASE-
> > > > >>> <https://issues.apache.org/jira/browse/HBASE-14123>14123
> > > > >>> Phase 3:
> > > > >>> https://issues.apache.org/jira/browse/HBASE-
> > > > >>> <https://issues.apache.org/jira/browse/HBASE-14414>14414
> > > > >>>
> > > > >>> -Vlad
> > > > >>>
> > > > >>> On Mon, Jul 11, 2016 at 4:41 PM, Vladimir Rodionov <
> > > > >> vladrodionov@gmail.com
> > > > >>> wrote:
> > > > >>>
> > > > >>>> These are the phases:
> > > > >>>>
> > > > >>>> Phase 1:
> > > > >>>> https://issues.apache.org/jira/browse/HBASE-
> > > > >>>> <https://issues.apache.org/jira/browse/HBASE-7912>14030
> > > > >>>> Phase 2:
> > > > >>>> https://issues.apache.org/jira/browse/HBASE-
> > > > >>>> <https://issues.apache.org/jira/browse/HBASE-7912>14123
> > > > >>>> Phase 3:
> > > > >>>> https://issues.apache.org/jira/browse/HBASE-
> > > > >>>> <https://issues.apache.org/jira/browse/HBASE-7912>14414
> > > > >>>>
> > > > >>>> -Vlad
> > > > >>>>
> > > > >>>>
> > > > >>>> On Mon, Jul 11, 2016 at 12:21 PM, Enis Söztutar <
> enis@apache.org>
> > > > >> wrote:
> > > > >>>>
> > > > >>>>> As you guys may already be familiar, Vladimir, Ted,
Jerry and
> > > others
> > > > >>> have
> > > > >>>>> been developing the backup / restore functionality
in a series
> of
> > > > >> issues
> > > > >>>>> committed in the separate branch HBASE-7912[1].
> > > > >>>>>
> > > > >>>>> Backup / Restore functionality is tracked as a 4-phase
project,
> > and
> > > > >> the
> > > > >>>>> first two phases are complete and useable. We are
now working
> on
> > > > >> Phase 3
> > > > >>>>> items, which are mostly improvements. We think that
the current
> > > code
> > > > >> in
> > > > >>>>> the
> > > > >>>>> branch containing all Phase 1 and Phase 2 items,
and some
> Phase 3
> > > > >> items
> > > > >>> is
> > > > >>>>> useable on it's own, and we do not have to wait for
all the
> > > > subtickets
> > > > >>> to
> > > > >>>>> be finished to make it completely useable (as follow
up tickets
> > are
> > > > >>> mostly
> > > > >>>>> improvements or optimizations). The improvements
in the works
> are
> > > all
> > > > >>>>> backwards compatible with the existing stuff. Thus,
we would
> like
> > > to
> > > > >>>>> propose that the branch HBASE-7912 be merged into
master.  The
> > > parent
> > > > >>> jira
> > > > >>>>> has a design doc that goes into details about the
> implementation
> > > and
> > > > >>>>> design
> > > > >>>>> choices in case you are interested[2].
> > > > >>>>>
> > > > >>>>> Most of the changes are largely non-intrusive and
confined to
> the
> > > > >>>>> backup subsystem.
> > > > >>>>> The unit tests have been passing on manual runs and
we
> > > (hortonworks)
> > > > >>> have
> > > > >>>>> been running the integration tests as well as some
other
> > > shell-based
> > > > >>>>> system
> > > > >>>>> tests on a forked version of the code. Most of the
work has
> been
> > > > >>> reviewed
> > > > >>>>> by 1, 2 or 3 committers already (mostly Ted, myself
and Jerry).
> > > > >>>>>
> > > > >>>>> What do you guys think? Is it time to call a vote?
Any concerns
> > or
> > > > >>>>> feedback
> > > > >>>>> appreciated.
> > > > >>>>>
> > > > >>>>> [1] https://issues.apache.org/jira/browse/HBASE-7912
> > > > >>>>> [2]
> > > > >>
> > > >
> > >
> >
> https://issues.apache.org/jira/secure/attachment/12816339/HBaseBackupAndRestore%20-0.91.pdf
> > > > >>>>>
> > > > >>>>> Enis
> > > > >>
> > > >
> > >
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message