incubator-cloudstack-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Kelven Yang <>
Subject Re: [PROPOSAL] BVT for CloudStack checkins
Date Wed, 06 Mar 2013 18:22:00 GMT
First +1 on BVT. 

Second, should we consider the idea of having a staging area for people to
check-in? Which is that making master always the stable(reasonable) branch
for main development, but whenever people make check-ins, it goes into
staging first, and we have maintainers(could be automatic) to run whatever
test framework we have and only perform automatic merge to master from
staging area after a successful test-run?


On 3/6/13 4:21 AM, "Prasanna Santhanam" <> wrote:

>Great to see you kick this off Alex! I have a few ideas for this and
>had been looking for help as well ..
>I had a draft lying around of an email you sent me long ago about
>tiered testing and I think your proposal fits very well on this:
>The tl;dr of that conversation was as below
>- improve simulator for runtime testability
>- customize to model and inject failures
>- make a habit of writing tests around bug reports
>(I started tagging tests since api_refactoring on JIRA already.
>look for the integration-test label)
>- make integration testing easier using factories and DSLs (from
>  Chiradeep)
>(part 1 of this work was started on this in the marvin-refactor
>More comments inline
>On Wed, Mar 06, 2013 at 12:12:11AM +0530, Alex Huang wrote:
>> Hi All,
>> As most of you are aware, the master branch keeps getting broken by
>> checkins for various reasons.  Committers need to be more
>> responsible about their checkins but I don't think we can depend on
>> that happening.  There are various reasons.  The most obvious to me
>> is that granting committership is not based on code competency.
>> (And I don't think it should.)  Given that we need to build a BVT
>> system for ensure that checkins do not break the branch.
>> Here's my proposal:
>> Existing components that we'll use.
>> -          Citrix has contributed its testing to Apache.
>> -          Apache CloudStack has already a simulator that's been used
>>for scale testing.
>> -          Marvin
>> -          DevCloud-kvm
>> Work Proposal:
>> -          Convert the Citrix testing into three phases:
>> o   Setup
>> o   Test
>> o   Verify
>I do the build, package, setup, test, verify in my integration test
>pipeline and a similar pipeline for a developer combined into easily
>runnable maven profiles/lifecycles/goals would be great to have.
>> -          Add a Setup and Verify phase for the simulator
>> -          Add all of the agent commands necessary for the simulator to
>>pass the testing.
>> -          Add a Setup and Verify phase for devCloud-kvm
>The setups exists as configs in the marvin sandbox already. Just need
>to hook it up with mvn. For verify there is a simple python script
> already that checks two things
>- system VMs are up
>- built-in templates are downloaded
>This should be a good start IMO.
>Currently devcloud-kvm is a bit hard to run from a developer
>environment. But it's great to have in a continuous environment backed
>by a KVM host with a Linux 3.0 kernel. Marcus has written up some
>pretty good documentation for this. If someone can help bring up that
>setup I can assist in bringing up the tests using my devcloud-ci
>scripts. I'm bringing up devcloud after Kelven 'alerted' us of the
>memory fix.
>> -          Add two more profiles to pom
>> o   Checkin-test-with-simulator: Runs the testing against the simulator
>> o   Checkin-test-with-devCloud: Runs the testing against devcloud
>> -          All of the profiles will attempt to also check the merge
>>list that Chip has proposed.'
>> -          We will also change marvin to easily add zones with
>> actual hardware.  It will be based on a data driven document to do
>> the setup.
>This is currently partly doable using a properties file in the sandbox
>$ python -i -o xen.cfg
>gives out a advanced zone config for the properties specified.
>Is data driven similar or you are talking about a DSL that Edison
>mentioned at CCC12?
>> For a developer to checkin:
>> -          S/he must writes marvin tests for their feature and add
>> it to the BVT.
>I made some progress on this in my marvin-refactor branch and will
>collect my thoughts in an FS that I am drafting. The marvin tests are
>difficult to write in their current form. I'm following Chiradeep's
>recommendation for providing factories and once those are baked a DSL
>language devs can quickly mock down tests against a simulator using
>the above mentioned mvn profiles.
>> -          S/he must run the checkin tests to verify everything works.
>> -          If the feature contains a hardware/vpx component, simulation
>>must be added.
>> At this point, everything is about the developer writing in their
>>feature branch and merging in.
>> On infrastructure side:
>> -          We'll setup continuous BVT based on the simulator.
>I brought this up on the IRC and the list y'day, so +1 - happy to help
>> -          I again push that we must use Gerrit to test the code
>> before it gets merge into the branch but I'll leave that for someone
>> else to do that.
>> Let me know what you guys think.  I'll probably break out a bvt
>> branch to work on this.  Anyone want to join me?
>Count me in!

View raw message