Mailing-List: contact cloudstack-dev-help@incubator.apache.org; run by ezmlm
Precedence: bulk
Reply-To: cloudstack-dev@incubator.apache.org
Received-SPF: pass (athena.apache.org: domain of kelven.yang@citrix.com
 designates 66.165.176.63 as permitted sender)
From: Kelven Yang <kelven.yang@citrix.com>
To: "cloudstack-dev@incubator.apache.org"
	<cloudstack-dev@incubator.apache.org>
Date: Wed, 6 Mar 2013 10:22:00 -0800
Subject: Re: [PROPOSAL] BVT for CloudStack checkins
Thread-Topic: [PROPOSAL] BVT for CloudStack checkins
Thread-Index: Ac4al38DOjYZtFmYS82wZzdZtChODg==
Message-ID: <CD5CC808.35E28%kelven.yang@citrix.com>
In-Reply-To: <20130306122142.GA12302@cloud-2.local>
Accept-Language: en-US
Content-Language: en-US
user-agent: Microsoft-MacOutlook/14.2.3.120616
acceptlanguage: en-US
Content-Type: text/plain; charset="us-ascii"
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0

First +1 on BVT.=20

Second, should we consider the idea of having a staging area for people to
check-in? Which is that making master always the stable(reasonable) branch
for main development, but whenever people make check-ins, it goes into
staging first, and we have maintainers(could be automatic) to run whatever
test framework we have and only perform automatic merge to master from
staging area after a successful test-run?

Kelven=20

On 3/6/13 4:21 AM, "Prasanna Santhanam" <tsp@apache.org> wrote:

>Great to see you kick this off Alex! I have a few ideas for this and
>had been looking for help as well ..
>
>I had a draft lying around of an email you sent me long ago about
>tiered testing and I think your proposal fits very well on this:
>
>The tl;dr of that conversation was as below
>
>- improve simulator for runtime testability
>- customize to model and inject failures
>- make a habit of writing tests around bug reports
>(I started tagging tests since api_refactoring on JIRA already.
>look for the integration-test label)
>- make integration testing easier using factories and DSLs (from
>  Chiradeep)
>(part 1 of this work was started on this in the marvin-refactor
>branch)
>
>More comments inline
>
>On Wed, Mar 06, 2013 at 12:12:11AM +0530, Alex Huang wrote:
>> Hi All,
>>=20
>> As most of you are aware, the master branch keeps getting broken by
>> checkins for various reasons.  Committers need to be more
>> responsible about their checkins but I don't think we can depend on
>> that happening.  There are various reasons.  The most obvious to me
>> is that granting committership is not based on code competency.
>> (And I don't think it should.)  Given that we need to build a BVT
>> system for ensure that checkins do not break the branch.
>>=20
>> Here's my proposal:
>>=20
>> Existing components that we'll use.
>>=20
>> -          Citrix has contributed its testing to Apache.
>>=20
>> -          Apache CloudStack has already a simulator that's been used
>>for scale testing.
>>=20
>> -          Marvin
>>=20
>> -          DevCloud-kvm
>>=20
>> Work Proposal:
>>=20
>> -          Convert the Citrix testing into three phases:
>>=20
>> o   Setup
>>=20
>> o   Test
>>=20
>> o   Verify
>
>I do the build, package, setup, test, verify in my integration test
>pipeline and a similar pipeline for a developer combined into easily
>runnable maven profiles/lifecycles/goals would be great to have.
>
>> -          Add a Setup and Verify phase for the simulator
>> -          Add all of the agent commands necessary for the simulator to
>>pass the testing.
>> -          Add a Setup and Verify phase for devCloud-kvm
>
>The setups exists as configs in the marvin sandbox already. Just need
>to hook it up with mvn. For verify there is a simple python script
>testSetupSuccess.py already that checks two things
>
>- system VMs are up
>- built-in templates are downloaded
>
>This should be a good start IMO.
>
>Currently devcloud-kvm is a bit hard to run from a developer
>environment. But it's great to have in a continuous environment backed
>by a KVM host with a Linux 3.0 kernel. Marcus has written up some
>pretty good documentation for this. If someone can help bring up that
>setup I can assist in bringing up the tests using my devcloud-ci
>scripts. I'm bringing up devcloud after Kelven 'alerted' us of the
>memory fix.
>
>>=20
>> -          Add two more profiles to pom
>> o   Checkin-test-with-simulator: Runs the testing against the simulator
>> o   Checkin-test-with-devCloud: Runs the testing against devcloud
>>=20
>> -          All of the profiles will attempt to also check the merge
>>list that Chip has proposed.'
>
>> -          We will also change marvin to easily add zones with
>> actual hardware.  It will be based on a data driven document to do
>> the setup.
>This is currently partly doable using a properties file in the sandbox
>$ python advanced_env.py -i xen.properties -o xen.cfg
>
>gives out a advanced zone config for the properties specified.
>Is data driven similar or you are talking about a DSL that Edison
>mentioned at CCC12?
>
>>=20
>> For a developer to checkin:
>>=20
>> -          S/he must writes marvin tests for their feature and add
>> it to the BVT.
>I made some progress on this in my marvin-refactor branch and will
>collect my thoughts in an FS that I am drafting. The marvin tests are
>difficult to write in their current form. I'm following Chiradeep's
>recommendation for providing factories and once those are baked a DSL
>language devs can quickly mock down tests against a simulator using
>the above mentioned mvn profiles.
>
>>=20
>> -          S/he must run the checkin tests to verify everything works.
>> -          If the feature contains a hardware/vpx component, simulation
>>must be added.
>>=20
>> At this point, everything is about the developer writing in their
>>feature branch and merging in.
>>=20
>> On infrastructure side:
>>=20
>> -          We'll setup continuous BVT based on the simulator.
>I brought this up on the IRC and the list y'day, so +1 - happy to help
>>=20
>> -          I again push that we must use Gerrit to test the code
>> before it gets merge into the branch but I'll leave that for someone
>> else to do that.
>>=20
>> Let me know what you guys think.  I'll probably break out a bvt
>> branch to work on this.  Anyone want to join me?
>
>Count me in!
>
>--=20
>Prasanna.,