hbase-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Vladimir Rodionov <vladrodio...@gmail.com>
Subject Re: [DISCUSSION] Merge Backup / Restore - Branch HBASE-7912
Date Sat, 09 Sep 2017 23:11:38 GMT
Hmm, the next on your list (of kicked out from branch v2) should be AM v2 I
presume?

-Vlad

On Sat, Sep 9, 2017 at 4:04 PM, stack <saint.ack@gmail.com> wrote:

> In spite of repeated requests for eng summary of state of this feature --
> summary of what is in 2.0, what is not, what the capabilities are, how well
> it has been tested and at what scale -- all I get, when the requests are
> not ignored, are pointers to lists of ill-describing jiras and some pending
> user facing doc update.
>
> For other features, mob or region server groups, I know that they have been
> running at scale in production for as much as a year and more. I have some
> confidence these items basically work.  For backup/restore I have no such
> sense even after spending time in review and trying to use the feature.
>
> As release manager, I have say over what makes it into a release.  Unless
> the work is done to convince me that backup/restore is more than a lump of
> code and a few unit tests that can pass on some fellows laptop, I am going
> to kick it out of branch-2.  Let the feature harden more in master branch
> before it ships in a release.
>
> S
>
> On Sep 8, 2017 10:59 PM, "Vladimir Rodionov" <vladrodionov@gmail.com>
> wrote:
>
> > >> Have I grasped the state of things correctly, Vlad?
> >
> > Josh, the only thing which is still pending is doc update. All other
> > features are good to have but not a blockers for 2.0 release.
> >
> > -Vlad
> >
> > On Fri, Sep 8, 2017 at 10:42 PM, Vladimir Rodionov <
> vladrodionov@gmail.com
> > >
> > wrote:
> >
> > > >> What testing and at what
> > > >> scale has testing been done?
> > >
> > > Do we have have that for other features?
> > >
> > >
> > > On Fri, Sep 8, 2017 at 10:41 PM, Vladimir Rodionov <
> > vladrodionov@gmail.com
> > > > wrote:
> > >
> > >> >> It asks: "How do I figure what of backup/restore feature is going
> to
> > >> be in
> > >> >>hbase-2.0.0?
> > >>
> > >> Hmm, wait for doc update.
> > >>
> > >>
> > >> On Fri, Sep 8, 2017 at 2:39 PM, Stack <stack@duboce.net> wrote:
> > >>
> > >>> HBASE-14414 is a JIRA with a list of random seeming issues w/
> > >>> non-descript
> > >>> summaries: "Add nonce support to TableBackupProcedure, BackupID must
> > >>> include backup set name, ...". The last comment in that issue is from
> > >>> July.
> > >>> It asks: "How do I figure what of backup/restore feature is going to
> be
> > >>> in
> > >>> hbase-2.0.0? Thanks Vladimir Rodionov
> > >>> <https://issues.apache.org/jira/secure/ViewProfile.jspa?
> name=vrodionov
> > >>> >."
> > >>> to which there is no answer.  Doc update is TODO.
> > >>>
> > >>> Where is the summary of the capability in hbase-2? What testing and
> at
> > >>> what
> > >>> scale has testing been done? Is this 'stable or experimental'? If I
> > can't
> > >>> get basic info on this feature though I ask repeatedly, what hope
> does
> > >>> the
> > >>> poor old operator have?
> > >>>
> > >>> St.Ack
> > >>>
> > >>>
> > >>> On Fri, Sep 8, 2017 at 1:59 PM, Vladimir Rodionov <
> > >>> vladrodionov@gmail.com>
> > >>> wrote:
> > >>>
> > >>> > HBASE-14414
> > >>> >
> > >>> > On Fri, Sep 8, 2017 at 1:14 PM, Stack <stack@duboce.net>
wrote:
> > >>> >
> > >>> > > Where do I go to get the current status of this feature?
Looking
> in
> > >>> JIRA
> > >>> > I
> > >>> > > see loads of issues open against backup including some against
> > >>> > hbase-2.0.0
> > >>> > > and no progress being made that I can discern.
> > >>> > >
> > >>> > > Thanks,
> > >>> > > S
> > >>> > >
> > >>> > >
> > >>> > >
> > >>> > > On Wed, Nov 23, 2016 at 8:52 AM, Stack <stack@duboce.net>
wrote:
> > >>> > >
> > >>> > > > On Tue, Nov 22, 2016 at 6:48 PM, Stack <stack@duboce.net>
> wrote:
> > >>> > > >
> > >>> > > >> On Tue, Nov 22, 2016 at 3:17 PM, Vladimir Rodionov
<
> > >>> > > >> vladrodionov@gmail.com> wrote:
> > >>> > > >>
> > >>> > > >>> >> and/or he answered most of the review
feedback
> > >>> > > >>>
> > >>> > > >>> No, questions are still open, but I do not see
any blockers
> and
> > >>> we
> > >>> > have
> > >>> > > >>> HBASE-16940 to address these questions.
> > >>> > > >>>
> > >>> > > >>>
> > >>> > > >> Agree. No blockers but stuff that should be dealt
with (No one
> > >>> will
> > >>> > pay
> > >>> > > >> me any attention once merge goes in -- smile).
> > >>> > > >>
> > >>> > > >>
> > >>> > > > Let me clarify the above. I want review addressed before
merge
> > >>> happens.
> > >>> > > > Sorry if any confusion.
> > >>> > > > St.Ack
> > >>> > > >
> > >>> > > >
> > >>> > > >
> > >>> > > >
> > >>> > > >
> > >>> > > >
> > >>> > > >> St.Ack
> > >>> > > >>
> > >>> > > >>
> > >>> > > >>
> > >>> > > >>> On Tue, Nov 22, 2016 at 3:04 PM, Devaraj Das
<
> > >>> ddas@hortonworks.com>
> > >>> > > >>> wrote:
> > >>> > > >>>
> > >>> > > >>> > Hi Stack, hats off to you for spending
so much time on
> this!
> > >>> > Thanks!
> > >>> > > >>> From
> > >>> > > >>> > my understanding, Vlad has raised follow-up
jiras for the
> > >>> issues
> > >>> > you
> > >>> > > >>> > raised, and/or he answered most of the
review feedback. So,
> > do
> > >>> you
> > >>> > > >>> think we
> > >>> > > >>> > could do a merge vote now?
> > >>> > > >>> > Devaraj.
> > >>> > > >>> > ________________________________________
> > >>> > > >>> > From: Vladimir Rodionov <vladrodionov@gmail.com>
> > >>> > > >>> > Sent: Monday, November 21, 2016 8:34 PM
> > >>> > > >>> > To: dev@hbase.apache.org
> > >>> > > >>> > Subject: Re: [DISCUSSION] Merge Backup
/ Restore - Branch
> > >>> > HBASE-7912
> > >>> > > >>> >
> > >>> > > >>> > >> I have spent a good bit of time
reviewing and testing
> this
> > >>> > > feature.
> > >>> > > >>> I
> > >>> > > >>> > would
> > >>> > > >>> > >> like my review and concerns addressed
and I'd like it to
> > be
> > >>> > clear
> > >>> > > >>> how;
> > >>> > > >>> > >> either explicit follow-on issues,
pointers to where in
> the
> > >>> patch
> > >>> > > or
> > >>> > > >>> doc
> > >>> > > >>> > my
> > >>> > > >>> > >> remarks have been catered to,
etc. Until then, I am
> > against
> > >>> > > commit.
> > >>> > > >>> >
> > >>> > > >>> > Stack, mega patch review comments will
be addressed in the
> > >>> > dedicated
> > >>> > > >>> JIRA:
> > >>> > > >>> > HBASE-16940
> > >>> > > >>> > I have open several other JIRAs to address
your other
> > comments
> > >>> (not
> > >>> > > on
> > >>> > > >>> > review board).
> > >>> > > >>> >
> > >>> > > >>> > Details are here (end of the thread):
> > >>> > > >>> > https://issues.apache.org/jira/browse/HBASE-14123
> > >>> > > >>> >
> > >>> > > >>> > Let me know what else should we do to move
merge forward.
> > >>> > > >>> >
> > >>> > > >>> > -Vlad
> > >>> > > >>> >
> > >>> > > >>> >
> > >>> > > >>> > On Fri, Nov 18, 2016 at 4:54 PM, Stack
<stack@duboce.net>
> > >>> wrote:
> > >>> > > >>> >
> > >>> > > >>> > > On Fri, Nov 18, 2016 at 3:53 PM, Ted
Yu <
> > yuzhihong@gmail.com
> > >>> >
> > >>> > > wrote:
> > >>> > > >>> > >
> > >>> > > >>> > > > Thanks, Matteo.
> > >>> > > >>> > > >
> > >>> > > >>> > > > bq. restore is not clear if given
an incremental id it
> > >>> will do
> > >>> > > the
> > >>> > > >>> full
> > >>> > > >>> > > > restore from full up to that
point or if i need to
> apply
> > >>> > manually
> > >>> > > >>> > > > everything
> > >>> > > >>> > > >
> > >>> > > >>> > > > The restore takes into consideration
of the dependent
> > >>> > backup(s).
> > >>> > > >>> > > > So there is no need to apply
preceding backup(s)
> > manually.
> > >>> > > >>> > > >
> > >>> > > >>> > > >
> > >>> > > >>> > > I ask this question on the issue.
It is not clear from
> the
> > >>> usage
> > >>> > or
> > >>> > > >>> doc
> > >>> > > >>> > how
> > >>> > > >>> > > to run a restore from incremental.
Can you fix in doc and
> > >>> usage
> > >>> > how
> > >>> > > >>> so I
> > >>> > > >>> > > can be clear and try it. Currently
I am stuck verifying a
> > >>> round
> > >>> > > trip
> > >>> > > >>> > backup
> > >>> > > >>> > > restore made of incrementals.
> > >>> > > >>> > >
> > >>> > > >>> > > Thanks,
> > >>> > > >>> > > S
> > >>> > > >>> > >
> > >>> > > >>> > >
> > >>> > > >>> > >
> > >>> > > >>> > > > On Fri, Nov 18, 2016 at 3:48
PM, Matteo Bertozzi <
> > >>> > > >>> > > theo.bertozzi@gmail.com>
> > >>> > > >>> > > > wrote:
> > >>> > > >>> > > >
> > >>> > > >>> > > > > I did one last pass to the
mega patch. I don't see
> > >>> anything
> > >>> > > major
> > >>> > > >>> > that
> > >>> > > >>> > > > > should block the merge.
> > >>> > > >>> > > > >
> > >>> > > >>> > > > > - most of the code is isolated
in the backup package
> > >>> > > >>> > > > > - all the backup code is
client side
> > >>> > > >>> > > > > - there are few changes
to the server side, mainly
> for
> > >>> > > cleaners,
> > >>> > > >>> wal
> > >>> > > >>> > > > > rolling and similar (which
is ok)
> > >>> > > >>> > > > > - there is a good number
of tests, and an integration
> > >>> test
> > >>> > > >>> > > > >
> > >>> > > >>> > > > > the code seems to have still
some left overs from the
> > old
> > >>> > > >>> > > implementation,
> > >>> > > >>> > > > > and some stuff needs a cleanup.
but I don't think
> this
> > >>> should
> > >>> > > be
> > >>> > > >>> used
> > >>> > > >>> > > as
> > >>> > > >>> > > > an
> > >>> > > >>> > > > > argument to block the merge.
I think the guys will
> keep
> > >>> > working
> > >>> > > >>> on
> > >>> > > >>> > this
> > >>> > > >>> > > > and
> > >>> > > >>> > > > > they may also get help of
others once the patch is in
> > >>> master.
> > >>> > > >>> > > > >
> > >>> > > >>> > > > > I still have my concerns
about the current
> limitations,
> > >>> but
> > >>> > > >>> these are
> > >>> > > >>> > > > > things already planned for
phase 3, so some of this
> > >>> stuff may
> > >>> > > >>> even be
> > >>> > > >>> > > in
> > >>> > > >>> > > > > the final 2.0.
> > >>> > > >>> > > > > but as long as we have a
"current limitations"
> section
> > >>> in the
> > >>> > > >>> user
> > >>> > > >>> > > guide
> > >>> > > >>> > > > > mentioning important stuff
like the ones below, I'm
> ok
> > >>> with
> > >>> > it.
> > >>> > > >>> > > > >  - if you write to the table
with
> Durability.SKIP_WALS
> > >>> your
> > >>> > > data
> > >>> > > >>> will
> > >>> > > >>> > > not
> > >>> > > >>> > > > > be in the incremental-backup
> > >>> > > >>> > > > >  - if you bulkload files
that data will not be in the
> > >>> > > incremental
> > >>> > > >>> > > backup
> > >>> > > >>> > > > > (HBASE-14417)
> > >>> > > >>> > > > >  - the incremental backup
will not only contains the
> > >>> data of
> > >>> > > the
> > >>> > > >>> > table
> > >>> > > >>> > > > you
> > >>> > > >>> > > > > specified but also the regions
from other tables that
> > >>> are on
> > >>> > > the
> > >>> > > >>> same
> > >>> > > >>> > > set
> > >>> > > >>> > > > > of RSs (HBASE-14141) ...maybe
a note about security
> > >>> around
> > >>> > this
> > >>> > > >>> topic
> > >>> > > >>> > > > >  - the incremental backup
will not contains just the
> > >>> "latest
> > >>> > > row"
> > >>> > > >>> > > between
> > >>> > > >>> > > > > backup A and B, but it will
also contains all the
> > updates
> > >>> > > >>> occurred in
> > >>> > > >>> > > > > between. but the restore
does not allow you to
> restore
> > >>> up to
> > >>> > a
> > >>> > > >>> > certain
> > >>> > > >>> > > > > point in time, the restore
will always be up to the
> > >>> "latest
> > >>> > > >>> backup
> > >>> > > >>> > > > point".
> > >>> > > >>> > > > >  - you should limit the
number of "incremental" up
> to N
> > >>> (or
> > >>> > > maybe
> > >>> > > >>> > > SIZE),
> > >>> > > >>> > > > to
> > >>> > > >>> > > > > avoid replay time becoming
the bottleneck.
> > (HBASE-14135)
> > >>> > > >>> > > > >
> > >>> > > >>> > > > > I'll be ok even with the
above not being in the final
> > >>> 2.0,
> > >>> > > >>> > > > > but i'd like to see as blocker
for the final 2.0 (not
> > the
> > >>> > > merge)
> > >>> > > >>> > > > >  - the backup code moved
in an hbase-backup module
> > >>> > > >>> > > > >  - and some more work around
tools, especially to try
> > to
> > >>> > unify
> > >>> > > >>> and
> > >>> > > >>> > make
> > >>> > > >>> > > > > simple the backup experience
(simple example: in some
> > >>> case
> > >>> > > there
> > >>> > > >>> is a
> > >>> > > >>> > > > > backup_id argument in others
a backupId argument. or
> > >>> things
> > >>> > > >>> like..
> > >>> > > >>> > > > restore
> > >>> > > >>> > > > > is not clear if given an
incremental id it will do
> the
> > >>> full
> > >>> > > >>> restore
> > >>> > > >>> > > from
> > >>> > > >>> > > > > full up to that point or
if i need to apply manually
> > >>> > > everything).
> > >>> > > >>> > > > >
> > >>> > > >>> > > > > in conclusion, I think we
can open a merge vote. I'll
> > be
> > >>> +1
> > >>> > on
> > >>> > > >>> it,
> > >>> > > >>> > and
> > >>> > > >>> > > I
> > >>> > > >>> > > > > think we should try to reject
-1 with just a "code
> > >>> cleanup"
> > >>> > > >>> > motivation,
> > >>> > > >>> > > > > since there will still be
work going on on the code
> > >>> after the
> > >>> > > >>> merge.
> > >>> > > >>> > > > >
> > >>> > > >>> > > > > Matteo
> > >>> > > >>> > > > >
> > >>> > > >>> > > > >
> > >>> > > >>> > > > > On Sun, Nov 6, 2016 at 10:54
PM, Devaraj Das <
> > >>> > > >>> ddas@hortonworks.com>
> > >>> > > >>> > > > wrote:
> > >>> > > >>> > > > >
> > >>> > > >>> > > > > > Stack and others, anything
else on the patch? Merge
> > to
> > >>> > master
> > >>> > > >>> now?
> > >>> > > >>> > > > > >
> > >>> > > >>> > > > >
> > >>> > > >>> > > >
> > >>> > > >>> > >
> > >>> > > >>> >
> > >>> > > >>>
> > >>> > > >>
> > >>> > > >>
> > >>> > > >
> > >>> > >
> > >>> >
> > >>>
> > >>
> > >>
> > >
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message