cloudstack-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Mike Tutkowski <mike.tutkow...@solidfire.com>
Subject Re: [PROPOSAL] move away from time-based releases and/or revamp release process
Date Tue, 24 Sep 2013 21:20:54 GMT
I think a distributed Jenkins setup would be great.

If we had really awesome test coverage, I would be less frightened of
last-minute checkins, as well. :)


On Tue, Sep 24, 2013 at 3:17 PM, Daan Hoogland <daan.hoogland@gmail.com>wrote:

> Mike, rest assured you and Marcus are not the only ones. More guarantee on
> a stable master is a general concern. Personally I don't feel we need more
> control on what is in the next release, if we make unit tests and automated
> integration tests a priority. That is kind of a claim I do have 'the'
> solution, though not well cooked ;) It's going to take a while (a colleague
> said four or five releases) before we have a good enough test set and a
> smoothly running continuous integration test engine. I think we at least
> need the distributed Jenkins setup where you can run your own integration
> tests to make sure your invested logic remains intact. This of course being
> only part of 'all the' answers.
>
> regards,
>
>
> On Tue, Sep 24, 2013 at 9:09 PM, Mike Tutkowski <
> mike.tutkowski@solidfire.com> wrote:
>
> > I was a bit hesitant to keep pushing this because there doesn't seem to
> be
> > a lot of support for it, but - as Marcus pointed out - I was quite
> alarmed
> > by the number and criticality of bugs checked in right before we cut our
> > first RC for 4.2. We simply were not ready.
> >
> > To me, it felt like something one might do before one gets out a decent
> > beta release.
> >
> > I certainly don't claim to have all the answers for this, but I do think
> we
> > need to develop some kind of a process whereby very few changes are made
> > immediately prior (like a month) to the first cut of a RC. We might even
> > need to discuss such changes as a community before they get checked in
> > (after a certain point).
> >
> > As far as master not always being usable, this is a serious problem, as
> > well.
> >
> > For example, I've been having trouble getting KVM to work and - in the
> > meanwhile - my code has fallen out of date with master over the past week
> > or so. However, I'm always afraid if I update from master while in the
> > middle of solving one problem that I'll have more problems to deal with
> > before I can get back to the initial problem (because something didn't
> work
> > in master).
> >
> > Again, I don't claim to have any solution for this problem, but I am
> happy
> > to help brainstorm.
> >
> >
> > On Tue, Sep 24, 2013 at 10:00 AM, Marcus Sorensen <shadowsor@gmail.com
> > >wrote:
> >
> > > On Mon, Sep 23, 2013 at 1:55 PM, Animesh Chaturvedi
> > > <animesh.chaturvedi@citrix.com> wrote:
> > > >
> > > >
> > > >> -----Original Message-----
> > > >> From: Marcus Sorensen [mailto:shadowsor@gmail.com]
> > > >> Sent: Monday, September 23, 2013 12:25 PM
> > > >> To: dev@cloudstack.apache.org
> > > >> Subject: RE: [PROPOSAL] move away from time-based releases and/or
> > revamp
> > > >> release process
> > > >>
> > > >> On Sep 23, 2013 1:03 PM, "Animesh Chaturvedi"
> > > >> <animesh.chaturvedi@citrix.com>
> > > >> wrote:
> > > >> >
> > > >> >
> > > >> >
> > > >> > > -----Original Message-----
> > > >> > > From: Marcus Sorensen [mailto:shadowsor@gmail.com]
> > > >> > > Sent: Monday, September 23, 2013 11:38 AM
> > > >> > > To: dev@cloudstack.apache.org
> > > >> > > Subject: [PROPOSAL] move away from time-based releases and/or
> > revamp
> > > >> > > release process
> > > >> > >
> > > >> > > Guys,  I think we are not currently in a state to handle
> > time-based
> > > >> > > releases.  Until we can cut master at any time and have
it
> > > >> > > releasable, or at least at a reasonable RC-level matching
> minimum
> > > >> > > tested requirements, it's just going to continue to be an
> exercise
> > > >> > > in frustration to cut RCs simply because we hit a deadline.
> > > >> > [Animesh>] David is going to propose Release Criterion up
for
> > > >> > discussion
> > > >> as per his thread [1]
> > > >>
> > > >> I see that thread more about defining what minimum bar we should
> > always
> > > >> have master at in order to meet time-based releases. Its where we
> want
> > > >> to go, but not what to do in the meantime.
> > > > [Animesh>] His proposal is not just for master, but also for deciding
> > > the release exit criterion and IMO is something we should follow for
> > 4.3.0
> > > and onwards
> > >
> > > Yes, I know. What I meant was that it will be a step toward
> > > stabilizing master, until we do that I'm not convinced we can adhere
> > > to any time-based expectation). It still doesn't fix our issue if
> > > we're going to insist on time-based releases, it just (from my
> > > undertanding) sets a bar for what is acceptable and what isn't, for
> > > any release. It stops the argument of "should we release with this
> > > bug".
> > >
> > > >>
> > > >> > >
> > > >> > > Maybe we can get away with sticking to time-based if we
revamp
> our
> > > >> > > schedule and procedures, I don't know, but in light of how
4.1
> > > >> > > (dragged on so long that some were seriously considering
> > > >> > > skipping/not releasing it with 4.2 on its heels) and 4.2
(six
> > rounds
> > > >> > > of votes so
> > > >> > > far) have worked it's probably worth discussing.
> > > >> > >
> > > >> > > Any suggestions on what might be better? It's been mentioned
in
> > the
> > > >> > > past that it's a chicken-egg thing, many really don't try
it
> until
> > > >> > > we hit an RC, which causes multiple iterations. I do agree
that
> > many
> > > >> > > don't take it seriously until we start cutting artifacts,
but
> > maybe
> > > >> > > we do this in a more deliberate fashion instead of jumping
right
> > to
> > > >> > > the vote. After feature/code freeze, cut some alpha artifacts,
> > wait
> > > >> > > a week, cut alpha2 or some beta artifacts, etc, and then
at some
> > > >> > > point anyone can propose that certain artifacts (or a new
set of
> > > >> > > artifacts) be put up for a vote as an RC. This gives us
a way to
> > > >> > > signal that we're gearing up for release and gives plenty
of
> time
> > > >> > > for people to test their components, or see the [PROPOSAL]
and
> say
> > > >> > > 'oh crap, I had better test my stuff', prior to cutting
an RC.
> > > >> > > Maybe this wouldn't help in practice, but I think right
now we
> go
> > > >> > > from telling the community "code is frozen, don't check
anything
> > in
> > > >> > > unless its a bug fix" to "here's our RC, try it out", without
a
> > > >> formal testing window.
> > > >> > > I realize the whole thing should be a testing window, but
I
> don't
> > > >> > > think it's conveyed well.
> > > >> >
> > > >> > [Animesh>] After the code freeze is all the stabilization
and
> > > >> > integration
> > > >> testing phase and has been documented at [2].  No one should be
> > waiting
> > > >> until the RC to test their components for the first time. It should
> be
> > > >> happening after code freeze.
> > > >>
> > > >> >
> > > >> > [1] http://markmail.org/thread/wlaq4zg36xnpgsjm
> > > >> > [2]
> https://cwiki.apache.org/confluence/display/CLOUDSTACK/Releases
> > > >> >
> > > >>
> > > >> Got it. As mentioned I realize that the whole time there is supposed
> > to
> > > >> be testing, but its not really working that way in practice. People
> > are
> > > >> volunteers, they forget where things are, or they dont want to mess
> > with
> > > >> it unless there is an indication that its semi-stable, and then
> > suddenly
> > > >> an RC is thrown over the fence and we go through iterations of RC.
> By
> > > >> the time the RC comes through we should be done testing and just
> > verify
> > > >> that someone's last minute bug fix didn't cause a regression or
> > > >> something.
> > > > [Animesh>] RC is not thrown in it is discussed as part of the release
> > > schedule.  After the code-freeze date everyone is expected to complete
> > > their integration testing by RC date. In fact I had sent numerous
> > reminders
> > > prior to the first RC starting from 2 weeks before the proposed RC
> date.
> > >
> > > That's not the point. The code is changing at a rapid pace. Mike, for
> > > example, commented on tons of critical fixes going in right up until
> > > the RC is cut. Then we cut some artifacts and give people 72 hours to
> > > test and buy off.   What I'm advocating is to lengthen the process,
> > > and not tie it to a timeline until we have better testing that
> > > stabilizes our master. At that time, when people can trust master
> > > remotely, then maybe individuals will take the time to poke at it
> > > prior to RC. Maybe that's a horrible idea, but let's at least talk
> > > about doing something until we're stable... or do we think we can
> > > accomplish that in a timely fashion?
> > >
> > > I think there are a few subgroups in our team here.  1) people whose
> > > job it is to develop on cloudstack, but don't really use it, 2) people
> > > who use cloudstack daily, and only do development to bugfix and/or add
> > > a pet feature. There may be some overlap for some individuals. This
> > > process might work great for individuals whose job it is to focus on
> > > cloudstack every single day and are tightly integrated with the
> > > massive changes, but the rest of us who consume cloudstack don't
> > > always have time to look at the big picture and focus on the unstable
> > > branches. We use the releases and focus on making the stable ones
> > > better and/or fixing/adding our pet features, until the next stable
> > > one comes around. Until the development branches stabilize I don't
> > > believe it will work for the users, they won't get involved until the
> > > end.
> > >
> > > For me, personally, it's a waste of time to even look at a branch that
> > > probably won't work due to sweeping changes that tend to occur between
> > > releases.  Make your core changes, add spring, replace the storage
> > > subsystem, whatever it is, and then I'll go back and see what it broke
> > > after the bugs are worked out in all of that. That's how group #2
> > > thinks, in general. And right now the only indicator that we're to
> > > that point is when we start talking RC, at which point I have a 3 day
> > > window that I hopefully catch and have time to play with it.
> > >
> > > >
> > > >
> > >
> > > My impression from your responses Animesh is that you feel everything
> > > is fine as-is.  I don't know how anyone could think that given what
> > > we've seen over the last two releases, especially you who had to cut
> > > six RCs. We're blowing past our "time based releases", and trying to
> > > push through buggy releases (for some reason). My intent was to sum up
> > > and focus on some of the comments I've seen over the past few weeks
> > > about low/sporadic RC participation, major changes going on at the
> > > last minute, etc. I guess I'm in the minority though, since we're the
> > > only ones discussing it.
> > >
> >
> >
> >
> > --
> > *Mike Tutkowski*
> > *Senior CloudStack Developer, SolidFire Inc.*
> > e: mike.tutkowski@solidfire.com
> > o: 303.746.7302
> > Advancing the way the world uses the
> > cloud<http://solidfire.com/solution/overview/?video=play>
> > *™*
> >
>



-- 
*Mike Tutkowski*
*Senior CloudStack Developer, SolidFire Inc.*
e: mike.tutkowski@solidfire.com
o: 303.746.7302
Advancing the way the world uses the
cloud<http://solidfire.com/solution/overview/?video=play>
*™*

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message