hbase-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jonathan Hsieh <...@cloudera.com>
Subject Re: Upcoming merge of snapshots branch into trunk. (HBASE-6055 and HABSE-7290)
Date Mon, 07 Jan 2013 18:58:27 GMT
On Mon, Jan 7, 2013 at 10:38 AM, Ted Yu <yuzhihong@gmail.com> wrote:

> Jon:
> Merging option #3 looks attractive. Do you have estimate for how long trunk
> should be frozen to other commits if we choose this approach ?
>
> The good news is that I don't think we actually need to freeze trunk.  S4
in the picture for #3 is something that can happen in parallel without
freezing trunk.  If changes in trunk that breaks the code in snapshots
(there currently are a few places), we'd do another S4-like commit (merge
trunk into snapshots, with review if non trivial) and then try to do the T4
commit again.  This is similar to the process used to merge various hdfs2
ha-branches into hdfs trunk.  Seemed like a good model to follow.

One catch is that  we've been in git -- and since my svn-fu is weaker than
git-fu, I'd have to spend some time figuring out how to do this with svn (I
know it is doable).


> Is snapshot-work-0103 the latest branch for HBASE-7290 repo ?
>
> The snapshot-work-0103 branch is the latest stuff that has alot of extra
code I've been keeping around in a compilable state, but not what I'd
merge.  Not all of it will go into the online merge.  Only the code in the
jyates/snapshtos (offline) or  jmhsieh/snapshots (offline+online) branch,
which has been reviewed and accepted is what I'd consider merging.  I'd bet
some of the commits afterwards don't compile (the last few reviews required
me to move code around between patches -- and I was only careful with what
was going to be committed).


> I ran Test*Snapshot* tests from this branch and found one failed test:
>
>
> testValidateGlobalSnapshotDescriptor(org.apache.hadoop.hbase.snapshot.TestSnapshotDescriptionUtils)
> Time elapsed: 0.112 sec  <<< ERROR!
> com.google.protobuf.UninitializedMessageException: Message missing required
> fields: name
>   at
>
> com.google.protobuf.AbstractMessage$Builder.newUninitializedMessageException(AbstractMessage.java:605)
>   at
>
> org.apache.hadoop.hbase.protobuf.generated.HBaseProtos$SnapshotDescription$Builder.build(HBaseProtos.java:11819)
>   at
>
> org.apache.hadoop.hbase.snapshot.TestSnapshotDescriptionUtils.testValidateGlobalSnapshotDescriptor(TestSnapshotDescriptionUtils.java:108)
>
>
This is safe to ignore for now -- I just intend to merge with the flush
online snapshot -- I don't intend to merge with the global online snapshot
or logroll online snapshot.  These last two would become full issues once
the branch has been merged.


> Cheers
>
> On Mon, Jan 7, 2013 at 10:07 AM, Jonathan Hsieh <jon@cloudera.com> wrote:
>
> > Hey Folks,
> >
> > Matteo, Jesse and I seem to be getting to the point where have core
> > functionality for offline snapshots (disable table, snapshot) and online
> > snapshot (snapshot an enabled table) committed, did another rev
> solidifying
> > file layout, and have been steadily knocking off blocking and
> non-blocking
> > follow-on subtasks.
> >
> > As a heads up, I think we are going to start considering a merge of the
> > snapshots branch to trunk. We agreed early on that we'd want 3 +1s before
> > the merge occurred.  Matteo, Jesse, and I worked on the core and all
> > committers now so we could technically satisfy this, but I'd really like
> to
> > have at least 1 reviewer look at the whole thing who didn't write parts
> of
> > it. :)  We'll also try to do another update on the design docs to help
> the
> > reviews.  Please consider taking a look at the branches and let us know.
> >
> > We're going to be spending this week hardening and testing are reasonable
> > scale to shake out more problems, so this may happen starting some time
> > next week.
> >
> > Aside from asking for reviews, there are a few outstanding questions we'd
> > love to get your feedback on:
> > HBASE-7471 - default configuration so that snapshots are available by
> > default?
> > HBASE-7360 - do we backport offline snapshots to the apache hbase 0.94
> > line?
> >
> > So where is the code and jiras? Currently, there are two branches -- an
> > offline snapshot only branch (HBASE-6055) here (
> > https://github.com/jyates/hbase/tree/snapshots aka
> > https://github.com/jmhsieh/hbase/tree/hbase-6055) and an offline +
> online
> > one (HBASE-7290) here
> > (https://github.com/jmhsieh/hbase/tree/snapshots).  Currently the
> > difference between the two are 3-4 patches (they are fairly substantial
> but
> > primarily additive).  We were being conservative initially and had hoped
> > that we could of done an offline merge earlier (and possibly merge with
> 94)
> > and then a second round when the online snapshot was more robust.  If our
> > testing goes well this week, for trunk I'd lean more towards just doing
> one
> > merge pass with the offline+online branch.
> >
> > Here's my current thoughts on the strategy and mechanics of the merge
> > (hopefully sometime next week).  Currently, our branches are based on a
> > 12/18 version of trunk. We're going to start doing merges of trunk into
> the
> > snapshot branch (to get all trunk patches into the snapshots).   I'll
> also
> > post a mega patch on review board after.  After we get all the reviews,
> we
> > could go a couple of ways, and would like your thoughts:
> >
> > 1) just commit the consolidated mega patch to trunk.  (every trunk step
> > should compile, but we lose blame information)
> > 2) rebase each patch and then a fix up patch at the end (yuck -- may have
> > broken intermediate steps, but keeps blame info)
> > 3) create a branch in svn (we've been in github) and, after we do an svn
> > merge of the snapshot branch into trunk. (more work, but each commit in
> > each branch should compile, keep blame information).
> >
> > #1 would look like this:
> >
> >    SS  SS is a single mega patch commit.
> >    |
> > S4 |   A merge of trunk into the snapshot branch (could be multiple)
> > | \|
> > |  T3  Trunk patch that when in while snapshots under dev.
> > S3 |
> > |  T2
> > S2 |
> > S1 |   Snapshot branch patches
> >   \|
> >    T1  The 12/18 point where snapshots was last rebased.
> >    ...
> >
> > #3 would be a bit of work but I think would be best in the end.   I think
> > the history would roughly look like this: (S is a snapshot commit, T is
> > trunk)
> >
> >    T4  Final merge commit into trunk. should be the same as S4
> >   /|
> > S4 |   A merge of trunk into the snapshot branch (could be multiple)
> > | \|
> > |  T3  Trunk patch that when in while snapshots under dev.
> > S3 |
> > |  T2
> > S2 |
> > S1 |   Snapshot branch patches
> >   \|
> >    T1  The 12/18 point where snapshots was last rebased.
> >    ...
> >
> > Comments, concerns?
> >
> > Thanks,
> > Jon.
> >
> > --
> > // Jonathan Hsieh (shay)
> > // Software Engineer, Cloudera
> > // jon@cloudera.com
> >
>



-- 
// Jonathan Hsieh (shay)
// Software Engineer, Cloudera
// jon@cloudera.com

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message