cloudstack-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From John Kinsella <...@stratosec.co>
Subject Re: Release cadence
Date Mon, 17 Mar 2014 16:52:52 GMT
I am in agreement with my radical CloudStack brother.


On Mar 13, 2014, at 9:42 AM, David Nalley <david@gnsa.us> wrote:

> The RC7 vote thread contained a lot of discussion around release
> cadence, and I figured I'd move that to a thread that has a better
> subject so there is better visibility to list participants who don't
> read every thread.
> 
> When I look at things schedule wise, I see our aims and our reality.
> We have a relatively short development window (in the schedule) and we
> have almost 50% of our time in the schedule allocated to testing.
> (over two months). However, it seems that a lot of testing - or at
> least a lot of testing for  what became blockers to the release didn't
> appear to happen until RCs were kicked out - and that's where our
> schedule has fallen apart for multiple releases. The automated tests
> we have were clean when we issued RCs, so we clearly don't have the
> depth needed from an automated standpoint.
> 
> Two problems, one cultural and one technical. The technical problem is
> that our automated test suite isn't deep enough to give us a high
> level of confidence that we should release. The cultural problem is
> that many of us wait until the release period of the schedule to test.
> 
> What does that have to do with release cadence? Well inherently not
> much; but let me describe my concerns. As a project; the schedule is
> meaningless if we don't follow it; and effectively the release date is
> held hostage. Personally, I do want as few bugs as possible, but it's
> a balancing act where people doubt our ability if we aren't able to
> ship. I don't think it matters if we move to 6 month cycles, if this
> behavior continues, we'd miss the 6 month date as well and push to 8
> or 9 months. See my radical proposition at the bottom for an idea on
> dealing with this.
> 
> I also find myself agreeing with Daan on the additional complexity.
> Increasing the window for release inherently increases the window for
> feature development. As soon as we branch a release, master is open
> for feature development again. This means a potential for greater
> change at each release. Change is a risk to quality; or at least an
> unknown that we again have to test. The greater that quantity of
> change, the greater the potential threat to quality.
> 
> Radical proposition:
> 
> Because we have two problems, of different nature, we are in a
> difficult situation. This is a possible solution, and I'd appreciate
> you reading and considering it.  Feedback is welcome. I propose that
> after we enter the RC stage that we not entertain any bugs as blockers
> that don't have automated test cases associated with them. This means
> that you are still welcome to do manual testing of your pet feature
> and the things that are important to you; during the testing window
> (or anytime really). However, if the automation suite isn't also
> failing then we consider the release as high enough quality to ship.
> This isn't something we can codify, but the PMC can certainly adopt
> this attitude as a group when voting. Which also means that we can
> deviate from it. If you brought up a blocker for release - we should
> be immediately looking at how we can write a test for that behavior.
> This should also mean several other behaviors need to become a valid
> part of our process. We need to ensure that things are well tested
> before allowing a merge. This means we need a known state of master,
> and we need to perform testing that allows us to confirm that a patch
> does no harm. We also need to insist on implementation of
> comprehensive tests for every inbound feature.
> 
> Thoughts, comments, flames, death threats? :)
> 
> --David


Mime
View raw message