cloudstack-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Prasanna Santhanam <>
Subject Re: [DISCUSS] QA/Testing focus on 4.1
Date Thu, 18 Oct 2012 14:23:00 GMT
On Wed, Oct 17, 2012 at 06:22:32PM -0400, David Nalley wrote:
> Warning: This is a long ponderous email. Stop now and get your cup
> of coffee/tea/$beverage.

I have my $beverage and I'm very happy to see this thread. :)

> So my proposal in short is that we focus on testing, beginning
> immediately and erect what is effectively a continuous deployment
> pipeline - such that our confidence in our codebase is such that we
> could arbitrarily decide to release if we so desired. I don't think
> that this is a one release type of project, indeed I think it's
> really more of a culture shift than a technical project. The big
> shift is that EVERYONE must be responsible for a quality release. To
> that end, I'd propose the following tenets if we choose to adopt
> this:

I agree completely. We need to stand up our infrastructure to provide
tests at all stages of the process - build, unit and integration.
Without which we only depend on our reviews of patches/fixes and a
sanity check to pass. That gets tedious and dangerous, if I may say,
since the project is a complex piece and things go bump at corner
cases unchecked.

> * Tests (specifically automated tests) become part of our culture.
> * New features should come replete with both unit and integration
> tests. I am tempted to suggest a certain percentage of coverage, but
> I worry that it is a red herring; particularly given our dismal
> current unit test coverage.

+1, along with Feature Specs being published I noticed reviews of the
feature and test plans too. This is great! This is also the best time
to define our tests and find the right coverage to give. My idea is we
start writing tests along side feature development so we don't wait
until the end. The coverage will naturally evolve over time.

>    * Blocker and critical bugs must have automated tests that get
>    committed as part of being qualified for closing.

+1000! I'm happy to track the bug reports to hunt for the ones that
can be automated and automate as much as possible.

> * We dedicate a non-trivial portion of our energy to enhancing not
> just the quantity and quality of our tests, but also on making it
> highly automated, and capable of delivering fast feedback. Ideally
> we would know within minutes if a commit broke unit tests, within
> hours if a commit failed in integration testing.

> Pipeline I'd like to see for 4.1:
> 1. RAT test (fail this and we have IP problems) 2. Compile test
> (does it actually compile) 3. Unit tests 4. Package building 5.
> Automated installation (multiple platforms, does it actually install
> from the packages) 6. Integration tests (aka Marvin running against
> virtualized or real CS deployments)

For 4) - We will need daily repos (yum/apt) for the builds to be
staged after they are packaged. Do we have a place to stage them now?
Something like and would be nice
to have.

For 5&6) - I've been trying to stand up infrastructure in a automated
fashion using puppet provisioning and combine that with Marvin tests.
I know this is a long time coming but it's been a little bit of a
struggle with the hotch-potch of systems that we've had to put
together and my first time with these tools.The nightly-smoke job on
jenkins is part of this work. Will detail the infrastructure setup for
this seperaetly once I have it running successfully ...

About Marvin tests - while there are a good number of them, will need
to expand coverage in the areas that haven't been touched. VPC tests
are being worked upon right now and with the features rolling out for
4.1 already on the lists we have more work to catch up on. 
> Clearly the above isn't an end/all be all for testing, but perhaps we
> can get some of this going in the 4.1 timeframe. There are also
> clearly corner cases (building marvin, building api docs, building
> official documentation) that need to be included in the pipeline as
> well. But the principle is that we won't move on past our failure
> until that failure is resolved.
> Immediate Action Items:
> Whether we adopt this or not, I plan on showing up on IRC once a week
> to work on testing in some form or another for an hour or two. I will
> also be cajoling people to join me. I might be working on
> infrastructure tasks. Obviously we have people scattered across the
> globe, so it's not the only time to work on testing, but you are
> welcome to join me.

Count me in! I would suggest we make it a weekly meeting on
#cloudstack-meeting. I see efforts from different folks - Yichi's
unittests for the API are much appreciated, some work underway on
jQuery tests for the AWS shim - we need to come together once a week
at least to discuss our progress going forward.

> I am curious to hear others thoughts, comments, or flames? Is this
> something we should espouse as we are close to 4.0.0 releasing and
> turning our focus on 4.1?
Timing is absolutely right. I sincerely hope all will be in place
before 4.1!


View raw message