cloudstack-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Prasanna Santhanam <>
Subject Re: [PROPOSAL] BVT for CloudStack checkins
Date Wed, 06 Mar 2013 12:21:42 GMT
Great to see you kick this off Alex! I have a few ideas for this and
had been looking for help as well ..

I had a draft lying around of an email you sent me long ago about
tiered testing and I think your proposal fits very well on this:

The tl;dr of that conversation was as below

- improve simulator for runtime testability 
- customize to model and inject failures
- make a habit of writing tests around bug reports 
(I started tagging tests since api_refactoring on JIRA already.
look for the integration-test label)
- make integration testing easier using factories and DSLs (from
(part 1 of this work was started on this in the marvin-refactor

More comments inline

On Wed, Mar 06, 2013 at 12:12:11AM +0530, Alex Huang wrote:
> Hi All,
> As most of you are aware, the master branch keeps getting broken by
> checkins for various reasons.  Committers need to be more
> responsible about their checkins but I don't think we can depend on
> that happening.  There are various reasons.  The most obvious to me
> is that granting committership is not based on code competency.
> (And I don't think it should.)  Given that we need to build a BVT
> system for ensure that checkins do not break the branch.
> Here's my proposal:
> Existing components that we'll use.
> -          Citrix has contributed its testing to Apache.
> -          Apache CloudStack has already a simulator that's been used for scale testing.
> -          Marvin
> -          DevCloud-kvm
> Work Proposal:
> -          Convert the Citrix testing into three phases:
> o   Setup
> o   Test
> o   Verify

I do the build, package, setup, test, verify in my integration test
pipeline and a similar pipeline for a developer combined into easily
runnable maven profiles/lifecycles/goals would be great to have.

> -          Add a Setup and Verify phase for the simulator
> -          Add all of the agent commands necessary for the simulator to pass the testing.
> -          Add a Setup and Verify phase for devCloud-kvm

The setups exists as configs in the marvin sandbox already. Just need
to hook it up with mvn. For verify there is a simple python script already that checks two things

- system VMs are up
- built-in templates are downloaded

This should be a good start IMO. 

Currently devcloud-kvm is a bit hard to run from a developer
environment. But it's great to have in a continuous environment backed
by a KVM host with a Linux 3.0 kernel. Marcus has written up some
pretty good documentation for this. If someone can help bring up that
setup I can assist in bringing up the tests using my devcloud-ci
scripts. I'm bringing up devcloud after Kelven 'alerted' us of the
memory fix.

> -          Add two more profiles to pom
> o   Checkin-test-with-simulator: Runs the testing against the simulator
> o   Checkin-test-with-devCloud: Runs the testing against devcloud
> -          All of the profiles will attempt to also check the merge list that Chip has

> -          We will also change marvin to easily add zones with
> actual hardware.  It will be based on a data driven document to do
> the setup.
This is currently partly doable using a properties file in the sandbox
$ python -i -o xen.cfg

gives out a advanced zone config for the properties specified.
Is data driven similar or you are talking about a DSL that Edison
mentioned at CCC12?

> For a developer to checkin:
> -          S/he must writes marvin tests for their feature and add
> it to the BVT.
I made some progress on this in my marvin-refactor branch and will
collect my thoughts in an FS that I am drafting. The marvin tests are
difficult to write in their current form. I'm following Chiradeep's
recommendation for providing factories and once those are baked a DSL
language devs can quickly mock down tests against a simulator using 
the above mentioned mvn profiles.

> -          S/he must run the checkin tests to verify everything works.
> -          If the feature contains a hardware/vpx component, simulation must be added.
> At this point, everything is about the developer writing in their feature branch and
merging in.
> On infrastructure side:
> -          We'll setup continuous BVT based on the simulator.
I brought this up on the IRC and the list y'day, so +1 - happy to help
> -          I again push that we must use Gerrit to test the code
> before it gets merge into the branch but I'll leave that for someone
> else to do that.
> Let me know what you guys think.  I'll probably break out a bvt
> branch to work on this.  Anyone want to join me?

Count me in!


View raw message