Mailing-List: contact cloudstack-dev-help@incubator.apache.org; run by ezmlm
Precedence: bulk
Reply-To: cloudstack-dev@incubator.apache.org
Received-SPF: pass (nike.apache.org: domain of sudha.ponnaganti@citrix.com
 designates 66.165.176.89 as permitted sender)
From: Sudha Ponnaganti <sudha.ponnaganti@citrix.com>
To: "cloudstack-dev@incubator.apache.org"
	<cloudstack-dev@incubator.apache.org>
Date: Wed, 17 Oct 2012 15:42:24 -0700
Subject: RE: [DISCUSS] QA/Testing focus on 4.1
Thread-Topic: [DISCUSS] QA/Testing focus on 4.1
Thread-Index: Ac2stgdFfYKY6QeJTfyvWFNqTRsnIQAATVCw
Message-ID: 
 <7914B38A4445B34AA16EB9F1352942F1012F109D6430@SJCPMAILBOX01.citrite.net>
References: 
 <CAKprHVaDa1nmofsrYM6M4dOPHTNNOXszAcvsXvuprAd++rHi4Q@mail.gmail.com>
In-Reply-To: 
 <CAKprHVaDa1nmofsrYM6M4dOPHTNNOXszAcvsXvuprAd++rHi4Q@mail.gmail.com>
Accept-Language: en-US
Content-Language: en-US
acceptlanguage: en-US
Content-Type: text/plain; charset="us-ascii"
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0

+1

We have covered may be 20-25% of overall coverage that should happen given =
the initial assessment on the available resources and timeline.=20
Going forward, definitely need to have automation in place for unit tests a=
nd regression which would help immensely.=20

If you can post timings on IRC, I will attend and see how I can participate=
 in these sessions and think of some strategies for overall quality standpo=
int


-----Original Message-----
From: David Nalley [mailto:david@gnsa.us]=20
Sent: Wednesday, October 17, 2012 3:23 PM
To: cloudstack-dev@incubator.apache.org
Subject: [DISCUSS] QA/Testing focus on 4.1

Warning: This is a long ponderous email. Stop now and get your cup of coffe=
e/tea/$beverage.

Hi folks,

I've been having some off hand conversations with folks in other open sourc=
e communities and rereading a number of good books of late (most notably Co=
ntinuous Delivery by Jez Humble) and I would like to toss this idea out and=
 see what folks think of it:

CloudStack is large, often complex, piece of software. There are few people=
 who truly grok all of CloudStack, and because of the diversity of environm=
ents it can be deployed in it can be truly bewildering to test fully. That =
said, I noticed a number of things that became last minute, blocker bugs, a=
nd it wasn't overly esoteric configurations, it was code paths that we just=
 hadn't excercised in testing. One if these would have even been easily det=
ected and fixed we had been testing installation alone. Additionally, as a =
project we are dramatically growing. The number of people adding non-trivia=
l amounts of code is growing, and that makes ad hoc QA even more difficult.

So my proposal in short is that we focus on testing, beginning immediately =
and erect what is effectively a continuous deployment pipeline - such that =
our confidence in our codebase is such that we could arbitrarily decide to =
release if we so desired. I don't think that this is a one release type of =
project, indeed I think it's really more of a culture shift than a technica=
l project. The big shift is that EVERYONE must be responsible for a quality=
 release. To that end, I'd propose the following tenets if we choose to ado=
pt this:

* Tests become the Andon cord[1] for the entire project. When a test fails =
we stop - additional commits don't happen - we find out what is wrong and f=
ix it. More on this in a bit.

* Tests (specifically automated tests) become part of our culture.
   * New features should come replete with both unit and integration tests.=
 I am tempted to suggest a certain percentage of coverage, but I worry that=
 it is a red herring; particularly given our dismal current unit test cover=
age.
   * Blocker and critical bugs must have automated tests that get committed=
 as part of being qualified for closing.

* We dedicate a non-trivial portion of our energy to enhancing not just the=
 quantity and quality of our tests, but also on making it highly automated,=
 and capable of delivering fast feedback. Ideally we would know within minu=
tes if a commit broke unit tests, within hours if a commit failed in integr=
ation testing.

I also know that this isn't a new idea. Lots of people have been focused on=
 automated testing as part of our ongoing development. The only difference =
here is that I am actively asking you not to solely depend on those folks t=
o do the work for you, but to make testing a part of the problem that you h=
ave to solve here. To be clear the goal isn't to be perfect and problem fre=
e with every commit - things will break. (If you've followed things at all =
in CloudStack you'll know that I've broken builds more than once)

Pipeline I'd like to see for 4.1:

1. RAT test (fail this and we have IP problems) 2. Compile test (does it ac=
tually compile) 3. Unit tests 4. Package building 5. Automated installation=
 (multiple platforms, does it actually install from the packages) 6. Integr=
ation tests (aka Marvin running against virtualized or real CS deployments)

Clearly the above isn't an end/all be all for testing, but perhaps we can g=
et some of this going in the 4.1 timeframe. There are also clearly corner c=
ases (building marvin, building api docs, building official documentation) =
that need to be included in the pipeline as well. But the principle is that=
 we won't move on past our failure until that failure is resolved.

Immediate Action Items:

Whether we adopt this or not, I plan on showing up on IRC once a week to wo=
rk on testing in some form or another for an hour or two. I will also be ca=
joling people to join me. I might be working on infrastructure tasks. Obvio=
usly we have people scattered across the globe, so it's not the only time t=
o work on testing, but you are welcome to join me.

I am curious to hear others thoughts, comments, or flames? Is this somethin=
g we should espouse as we are close to 4.0.0 releasing and turning our focu=
s on 4.1?

--David

[1] http://en.wikipedia.org/wiki/Andon_(manufacturing)