Mailing-List: contact hdfs-dev-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
MIME-Version: 1.0
In-Reply-To: <CAGB5D2bdcDaBE75TAFj2S27aMYQ2pR-hOA_NXmcaRb8wratUMg@mail.gmail.com>
References: <CAGB5D2Z_bb_1KrRRp-rDey-aXd19vpJ14xSD+A8k_zJfbQqcNA@mail.gmail.com>
	<CALwhT97fpD-L9VB3oQN08LNNKhf9wcDiFTytiRRr1Wk_k0aCHg@mail.gmail.com>
	<1465567183392.4149@hortonworks.com>
	<CAMXuLLq_-hWgYxreaAYkLj43xhAMh4LJuMs9LtaZdJbpi-BLSg@mail.gmail.com>
	<CADbBEntV_9KtrXjPhBz+FBoLa2EBsXZK3LhiqjWANtFXNizAjg@mail.gmail.com>
	<BD5F83F9-2EF7-4CD0-B4E8-65449A3D0BE9@hortonworks.com>
	<CADbBEnu0OP8agpC30UjvVSFkhcv-rBe4=Zj00WFRsu59Dgh1Rw@mail.gmail.com>
	<CAGB5D2bdcDaBE75TAFj2S27aMYQ2pR-hOA_NXmcaRb8wratUMg@mail.gmail.com>
Date: Fri, 10 Jun 2016 21:39:21 -0700
Message-ID: <CALwhT96HcCti=thWcVCBTOiSE1+GOMLuh0T2NM+PbkkSctsxHg@mail.gmail.com>
Subject: Re: [DISCUSS] Increased use of feature branches
From: Karthik Kambatla <kasha@cloudera.com>
To: Andrew Wang <andrew.wang@cloudera.com>
Cc: Sangjin Lee <sjlee@apache.org>, Anu Engineer <aengineer@hortonworks.com>,
	Zhe Zhang <zhz@apache.org>, Junping Du <jdu@hortonworks.com>,
	"common-dev@hadoop.apache.org" <common-dev@hadoop.apache.org>,
	"hdfs-dev@hadoop.apache.org" <hdfs-dev@hadoop.apache.org>,
	"mapreduce-dev@hadoop.apache.org" <mapreduce-dev@hadoop.apache.org>,
	"yarn-dev@hadoop.apache.org" <yarn-dev@hadoop.apache.org>
Content-Type: multipart/alternative; boundary=001a1149333c2a67710534f93d85
archived-at: Sat, 11 Jun 2016 04:39:36 -0000

--001a1149333c2a67710534f93d85
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: quoted-printable

I would like to understand the trunk-incompat part of the proposal a little
better.

Is trunk-incompat always going to be a superset of trunk? If yes, is it
just a change in naming convention with a hope that our approach to trunk
stability changes as Sangjin mentioned?

Or, is it okay for trunk-incompat to be based off of an older commit in
trunk with (in)frequent rebases? This has the risk of incompatible changes
truly rotting. Periodic rebases will ensure these changes don't rot while
also easing the burden of hosting two branches; if we choose this route,
some guidance of the period and who rebases will be nice.

On Fri, Jun 10, 2016 at 5:11 PM, Andrew Wang <andrew.wang@cloudera.com>
wrote:

> Let me try to clarify a few points, since not everyone might have been
> present for the previous emails.
>
> On the "Looking to a Hadoop 3 release" thread, we already reached
> consensus on doing releases from trunk. People didn't want to have to
> commit to another branch, and wanted to try releasing from trunk. The
> question, then, was how to ensure that trunk remains stable and releasabl=
e.
>
> Part of Vinod's proposal was that we, as a community, be more judicious
> about what we commit to trunk, and try to make use of more feature branch=
es
> for larger efforts. There was no requirement that 1-2 patch changes go
> through a feature branch. There weren't any requirements around # of
> patches or length of development at all, just asking that committers be
> more judicious. I personally think Sangjin's rule of thumb of ~12 patches
> or ~1 month are about right, but it's up to the developers who are
> involved, and I doubt any one standard will fit all situations.
>
> So, this is about as low-overhead a policy there is: devs, please be
> careful when committing to trunk, and consider using a feature branch for
> bigger efforts.
>
> If you have further ideas about how to improve stability of trunk, I'd
> love to hear it. I'd hope though that the above would be a
> non-controversial statement.
>
> Best,
> Andrew
>
> On Fri, Jun 10, 2016 at 2:10 PM, Sangjin Lee <sjlee@apache.org> wrote:
>
>> Thanks for your thoughts Anu.
>>
>> Regarding your question
>>
>>> And then comes the question, once 3.0 becomes official, where do we
>>> check-in a change,  if that would break something? so this will lead us
>>> back to trunk being the unstable =E2=80=93 3.0 being the new =E2=80=9Cb=
ranch-2=E2=80=9D.
>>
>>
>> Andrew mentioned in the original email
>>
>>> Regarding "trunk-incompat", since we're still in the alpha stage for
>>> 3.0.0, there's no need for this branch yet. This aspect of Vinod's prop=
osal
>>> was still under a bit of discussion; Chris Douglas though we should cut=
 a
>>> branch-3 for the first 3.0.0 beta, which aligns with my original thinki=
ng.
>>> This point doesn't necessarily need to be resolved now though, since ag=
ain
>>> we're still doing alphas.
>>
>>
>> and I agree with that sentiment. I think even if we have a
>> "trunk-incompat" branch to hold future incompatible changes, the situati=
on
>> will change little from today. Instead of dealing with "trunk" (where
>> incompatible changes may appear) and "branch-3", we would be dealing wit=
h
>> "trunk-incompat" and "trunk". Names are largely mnemonics then.
>>
>>
>> On Fri, Jun 10, 2016 at 12:37 PM, Anu Engineer <aengineer@hortonworks.co=
m
>> > wrote:
>>
>>> I actively work on two branches (Diskbalancer and ozone) and I agree
>>> with most of what Sangjin said.
>>> There is an overhead in working with branches, there are both technical
>>> costs and administrative issues
>>> which discourages developers from using branches.
>>>
>>> I think the biggest issue with branch based development is that fact
>>> that other developers do not use a branch.
>>> If a small feature appears as a series of commits to =E2=80=9C=E2=80=9D=
datanode.java=E2=80=9D=E2=80=9D,
>>> the branch based developer ends up rebasing
>>> and paying this price of rebasing many times. If everyone followed a
>>> model of branch + Pull request, other branches
>>> would not have to deal with continues rebasing to trunk commits. If we
>>> are moving to a branch based
>>> development, we should probably move to that model for most development
>>> to avoid this tax on people who
>>>  actually end up working in the branches.
>>>
>>> I do have a question in my mind though: What is being proposed is that
>>> we move active development to branches
>>> if the feature is small or incomplete, however keep the trunk open for
>>> check-ins. One of the biggest reason why we
>>> check-in into trunk and not to branch-2 is because it is a change that
>>> will break backward compatibility. So do we
>>> have an expectation of backward compatibility thru the 3.0-alpha series
>>> (I personally vote No, since 3.0 is experimental
>>> at this stage), but if we decide to support some sort of
>>> backward-compact then willy-nilly committing to trunk
>>> and still maintaining the expectation we can release Alphas from 3.0
>>> does not look possible.
>>>
>>> And then comes the question, once 3.0 becomes official, where do we
>>> check-in a change,  if that would break something?
>>> so this will lead us back to trunk being the unstable =E2=80=93 3.0 bei=
ng the
>>> new =E2=80=9Cbranch-2=E2=80=9D.
>>>
>>> One more point: If we are moving to use a branch always =E2=80=93 then =
we are
>>> looking at a model similar to using a git + pull
>>> request model. If that is so would it make sense to modify the rules to
>>> make these branches easier to merge?
>>> Say for example, if all commits in a branch has followed review and
>>> checking policy =E2=80=93 just like trunk and commits
>>> have been made only after a sign off from a committer, would it be
>>> possible to merge with a 3-day voting period
>>> instead of 7, or treat it just like today=E2=80=99s commit to trunk =E2=
=80=93 but with 2
>>> people signing-off?
>>>
>>> What I am suggesting is reducing the administrative overheads of using =
a
>>> branch to encourage use of branching.
>>> Right now it feels like Apache=E2=80=99s process encourages committing =
directly
>>> to trunk than a branch
>>>
>>> Thanks
>>> Anu
>>>
>>>
>>> On 6/10/16, 10:50 AM, "sjlee0@gmail.com on behalf of Sangjin Lee" <
>>> sjlee0@gmail.com on behalf of sjlee@apache.org> wrote:
>>>
>>> >Having worked on a major feature in a feature branch, I have some
>>> thoughts
>>> >and observations on feature branch development.
>>> >
>>> >IMO feature branch development v. direct commits to trunk in piecemeal
>>> is
>>> >really a choice of *granularity*. Do we want a series of fine-grained
>>> state
>>> >changes on trunk or fewer coarse-grained chunks of commits on trunk?
>>> >
>>> >This makes me favor a branch-based development model for any
>>> "decent-sized"
>>> >features (we'll need to define "decent-sized" of course). Once you hav=
e
>>> >coarse-grained changes, it's easier to reason about what made what
>>> release
>>> >and in what state. As importantly, it makes it easier to back out a
>>> >complete feature fairly easily if that becomes necessary. My totally
>>> >unscientific suggestion may be if a feature takes more than dozen
>>> commits
>>> >and longer than a month, we should probably have a bias towards a
>>> feature
>>> >branch.
>>> >
>>> >Branch-based development also makes you go faster if your feature is
>>> >larger. I wouldn't do it the other way for timeline service v.2 for
>>> example.
>>> >
>>> >That said, feature branches don't come for free. Now the onus is on th=
e
>>> >feature developer to constantly rebase with the trunk to keep it
>>> reasonably
>>> >integrated with the trunk. More logistics is involved for the feature
>>> >developer. Another big question is, when a feature branch gets big and
>>> it's
>>> >time to merge, would it get as scrutinized as a series of individual
>>> >commits? Since the size of merge can be big, you kind of have to rely =
on
>>> >those feature committers and those who help them.
>>> >
>>> >In terms of integrating/stabilizing, I don't think branch development
>>> >necessarily makes it harder. It is again granularity. In case of direc=
t
>>> >commits on trunk, you do a lot more fine-grained integrations. In case
>>> of
>>> >branch development, you do far fewer coarse-grained integrations via
>>> >rebasing. If more people are doing branch-based development, it makes
>>> >rebasing easier to manage too.
>>> >
>>> >Going back to the related topic of where to release (trunk v.
>>> branch-X), I
>>> >think that is more of a proxy of the real question of "how do we
>>> maintain
>>> >quality and stability of the trunk?". Even if we release from the
>>> trunk, if
>>> >our bar for merging to trunk is low, the quality will not improve
>>> >automatically. So I think we ought to tackle the quality question firs=
t.
>>> >
>>> >My 2 cents.
>>> >
>>> >
>>> >On Fri, Jun 10, 2016 at 8:57 AM, Zhe Zhang <zhz@apache.org> wrote:
>>> >
>>> >> Thanks for the notes Andrew, Junping, Karthik.
>>> >>
>>> >> Here are some of my understandings:
>>> >>
>>> >> - Trunk is the "latest and greatest" of Hadoop. If a user starts usi=
ng
>>> >> Hadoop today, without legacy workloads, trunk is what he/she should
>>> use.
>>> >> - Therefore, each commit to trunk should be transactional -- atomic,
>>> >> consistent, isolated (from other uncommitted patches); I'm not so su=
re
>>> >> about durability, Hadoop might be gone in 50 years :). As a
>>> committer, I
>>> >> should be able to look at a patch and determine whether it's a
>>> >> self-contained improvement of trunk, without looking at other
>>> uncommitted
>>> >> patches.
>>> >> - Some comments inline:
>>> >>
>>> >> On Fri, Jun 10, 2016 at 6:56 AM Junping Du <jdu@hortonworks.com>
>>> wrote:
>>> >>
>>> >> > Comparing with advantages, I believe the disadvantages of shipping
>>> any
>>> >> > releases directly from trunk are more obvious and significant:
>>> >> > - A lot of commits (incompatible, risky, uncompleted feature, etc.=
)
>>> have
>>> >> > to wait to commit to trunk or put into a separated branch that cou=
ld
>>> >> delay
>>> >> > feature development progress as additional vote process get
>>> involved even
>>> >> > the feature is simple and harmless.
>>> >> >
>>> >> Thanks Junping, those are valid concerns. I think we should clearly
>>> >> separate incompatible with  uncompleted / half-done work in this
>>> >> discussion. Whether people should commit incompatible changes to
>>> trunk is a
>>> >> much more tricky question (related to trunk-incompat etc.). But per =
my
>>> >> comment above, IMHO, *not committing uncompleted work to trunk*
>>> should be a
>>> >> much easier principle to agree upon.
>>> >>
>>> >>
>>> >> > - For small feature with only 1 or 2 commits, that need three +1
>>> from
>>> >> PMCs
>>> >> > will increase the bar largely for contributors who just start to
>>> >> contribute
>>> >> > on Hadoop features but no such sufficient support.
>>> >> >
>>> >> Development overhead is another valid concern. I think our
>>> rule-of-thumb
>>> >> should be that, small-medium new features should be proposed as a
>>> single
>>> >> JIRA/patch (as we recently did for HADOOP-12666). If the complexity
>>> goes
>>> >> beyond a single JIRA/patch, use a feature branch.
>>> >>
>>> >>
>>> >> >
>>> >> > Given these concerns, I am open to other options, like: proposed b=
y
>>> Vinod
>>> >> > or Chris, but rather than to release anything directly from trunk.
>>> >> >
>>> >> > - This point doesn't necessarily need to be resolved now though,
>>> since
>>> >> > again we're still doing alphas.
>>> >> > No. I think we have to settle down this first. Without a common
>>> agreed
>>> >> and
>>> >> > transparent release process and branches in community, any release
>>> >> (alpha,
>>> >> > beta) bits is only called a private release but not a official
>>> apache
>>> >> > hadoop release (even alpha).
>>> >> >
>>> >> >
>>> >> > Thanks,
>>> >> >
>>> >> > Junping
>>> >> > ________________________________________
>>> >> > From: Karthik Kambatla <kasha@cloudera.com>
>>> >> > Sent: Friday, June 10, 2016 7:49 AM
>>> >> > To: Andrew Wang
>>> >> > Cc: common-dev@hadoop.apache.org; hdfs-dev@hadoop.apache.org;
>>> >> > mapreduce-dev@hadoop.apache.org; yarn-dev@hadoop.apache.org
>>> >> > Subject: Re: [DISCUSS] Increased use of feature branches
>>> >> >
>>> >> > Thanks for restarting this thread Andrew. I really hope we can get
>>> this
>>> >> > across to a VOTE so it is clear.
>>> >> >
>>> >> > I see a few advantages shipping from trunk:
>>> >> >
>>> >> >    - The lack of need for one additional backport each time.
>>> >> >    - Feature rot in trunk
>>> >> >
>>> >> > Instead of creating branch-3, I recommend creating branch-3.x so w=
e
>>> can
>>> >> > continue doing 3.x releases off branch-3 even after we move trunk
>>> to 4.x
>>> >> (I
>>> >> > said it :))
>>> >> >
>>> >> > On Thu, Jun 9, 2016 at 11:12 PM, Andrew Wang <
>>> andrew.wang@cloudera.com>
>>> >> > wrote:
>>> >> >
>>> >> > > Hi all,
>>> >> > >
>>> >> > > On a separate thread, a question was raised about 3.x branching
>>> and use
>>> >> > of
>>> >> > > feature branches going forward.
>>> >> > >
>>> >> > > We discussed this previously on the "Looking to a Hadoop 3
>>> release"
>>> >> > thread
>>> >> > > that has spanned the years, with Vinod making this proposal
>>> (building
>>> >> on
>>> >> > > ideas from others who also commented in the email thread):
>>> >> > >
>>> >> > >
>>> >> > >
>>> >> >
>>> >>
>>> http://mail-archives.apache.org/mod_mbox/hadoop-common-dev/201604.mbox/=
browser
>>> >> > >
>>> >> > > Pasting here for ease:
>>> >> > >
>>> >> > > On an unrelated note, offline I was pitching to a bunch of
>>> >> > > contributors another idea to deal
>>> >> > > with rotting trunk post 3.x: *Make 3.x releases off of trunk
>>> directly*.
>>> >> > >
>>> >> > > What this gains us is that
>>> >> > >  - Trunk is always nearly stable or nearly ready for releases
>>> >> > >  - We no longer have some code lying around in some branch
>>> (today=E2=80=99s
>>> >> > > trunk) that is not releasable
>>> >> > > because it gets mixed with other undesirable and incompatible
>>> changes.
>>> >> > >  - This needs to be coupled with more discipline on individual
>>> >> > > features - medium to to large
>>> >> > > features are always worked upon in branches and get merged into
>>> trunk
>>> >> > > (and a nearing release!)
>>> >> > > when they are ready
>>> >> > >  - All incompatible changes go into some sort of a trunk-incompa=
t
>>> >> > > branch and stay there till
>>> >> > > we accumulate enough of those to warrant another major release.
>>> >> > >
>>> >> > > Regarding "trunk-incompat", since we're still in the alpha stage
>>> for
>>> >> > 3.0.0,
>>> >> > > there's no need for this branch yet. This aspect of Vinod's
>>> proposal
>>> >> was
>>> >> > > still under a bit of discussion; Chris Douglas though we should
>>> cut a
>>> >> > > branch-3 for the first 3.0.0 beta, which aligns with my original
>>> >> > thinking.
>>> >> > > This point doesn't necessarily need to be resolved now though,
>>> since
>>> >> > again
>>> >> > > we're still doing alphas.
>>> >> > >
>>> >> > > What we should get consensus on is the goal of keeping trunk
>>> stable,
>>> >> and
>>> >> > > achieving that by doing more development on feature branches and
>>> being
>>> >> > > judicious about merges. My sense from the Hadoop 3 email thread
>>> (and
>>> >> the
>>> >> > > more recent one on the async API) is that people are generally i=
n
>>> favor
>>> >> > of
>>> >> > > this.
>>> >> > >
>>> >> > > We're just about ready to do the first 3.0.0 alpha, so would
>>> greatly
>>> >> > > appreciate everyone's timely response in this matter.
>>> >> > >
>>> >> > > Thanks,
>>> >> > > Andrew
>>> >> > >
>>> >> >
>>> >> >
>>> ---------------------------------------------------------------------
>>> >> > To unsubscribe, e-mail: common-dev-unsubscribe@hadoop.apache.org
>>> >> > For additional commands, e-mail: common-dev-help@hadoop.apache.org
>>> >> >
>>> >> >
>>> >>
>>>
>>>
>>
>

--001a1149333c2a67710534f93d85--