cloudstack-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Erik Weber <>
Subject Re: Minor releases!
Date Thu, 07 Jan 2016 14:04:34 GMT
On Thu, Jan 7, 2016 at 2:28 PM, Remi Bergsma <>

> Hi René, all,
> I simply don’t understand why you need lots and lots of minor versions. I
> do understand you need a stable cloud, and that’s exactly what we’re
> achieving here.
> We changed our way of working from 4.6 on. Before that, it took _a long_
> time to release new versions (be it major, minor or patch). Releases were
> not high quality so people waited for .1 or .2 to be released. When you did
> eventually upgrade, you’d hit major trouble. It was simply not OK. And many
> minor releases don’t help here. The root cause of the trouble is that the
> whole idea of branching off a new version and have a QA team make it stable
> (while the rest continues on master) doesn’t make sense (any more). That’s
> why in the summer of 2015 we decided to stabilise master instead and
> release from that. [1] One final time it took a great effort to make a
> branch stable.
> We released 4.6 in Nov 2015
> We released 4.7 in Dec 2015
> We will release 4.8 in Jan 2016 (unless people think I should stop doing
> this)
> In between, we submitted patch releases to quickly address bugs (4.6.1,
> 4.6.2 etc).
> You know what? It’s easy to do these upgrades. Much much easier than
> before. The change is small, the procedure is quick. I upgraded our
> production environment from 4.6.2 to 4.7.0 in under 10 minutes (a clustered
> setup). When 4.7.0 got released, for the first time ever, we knew 100% sure
> it included everything from 4.6 as it was built on top of it. No regression
> between 4.6 and 4.7. Actually there’s no need to use 4.6 any more, when 4.7
> is out. It’s essentially the same, plus new features.
> So, yes, monthly releases can be done and the quality is better than
> before. Actually, I think we should go much faster. Whenever a PR is merged
> that fixes your issue, it should be possible to deploy it right away. It’s
> a change in mindset.
> Before this change, master was broken all the time. Now you can even run
> production from master. It just works (tm). Users should pick the latest
> version.
> If someone wants to maintain an older release, that can be done. I know
> Rohit maintains older versions for ShapeBlue customers and that makes
> sense. Until everyone gets used to an agile way of working this may still
> be needed. I just don’t work that way any more.
> To summarise, you want a stable cloud and I think since 4.6 we provide
> just that. Quick releases, fast delivery of new features and even faster
> delivery of bug fixes. There will always be bugs, so the faster we release,
> the smaller the diff and the easier the deploy and best of all: the sooner
> the bug is resolved in production.
> To be honest, I have the feeling not much people agree with me. So maybe I
> should stop doing this. I also see it in the number of people reviewing and
> testing. It’s simply disappointing. Anyway, let me know what you guys want.
> If people want to go back to the old way of working then that’s also fine
> with me. I just won’t be the Release Manager of it.

I don't think people necessarily disagree with you, but they have/come from
a different background.
You are heavily involved with the development of CloudStack, a lot of our
users (including me) are not.

And even if 4.6 and 4.7 is considered stable, and testing efforts have
increased dramastically, there is always a chance for regressions.
Just take a look at all the VR fixes that has come in after 4.6 (I am not
saying they are all regressions!)

I did try a 4.6 upgrade from 4.5 which essentially broke a lot of our VPC
usage and had to revert.
As a user I have to account for that before doing the upgrade, which means
I have to do it in a maintenance window.

With minor releases the risk is less, because there's no new features, just
bug fixes (and possible improvements?), and that is easier to account for
if you don't really need the new features right away.

Another note on the rapid release schedule is that it stresses testing
efforts, you already scream about more testers, but essentially testing is
a 1-week job every month/release, not everyone have time and resources for

To summarize; I (and hopefully most other users) really appreciate all the
hard work done by SBP, SB, Leasweb, PCextreme +++, but not everyone is in
the same positition to do rapid upgrades as you guys are.

Let me know!
> Regards,
> Remi
> [1]
> PS: At work we run multiple CloudStack powered clouds with hundreds of
> hypervisors so I think I know what I'm talking about.
Nobody questioned your skills, but they are not you. And for someone with a
single install with 2 hypervisors they don't necessarily have the same
amount of resources available.


  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message