Mailing-List: contact cloudstack-dev-help@incubator.apache.org; run by ezmlm
Precedence: bulk
Reply-To: cloudstack-dev@incubator.apache.org
Received-SPF: pass (athena.apache.org: domain of
 Simon.Waterhouse@eu.citrix.com designates 46.33.159.39 as permitted sender)
From: Simon Waterhouse <Simon.Waterhouse@eu.citrix.com>
To: "cloudstack-dev@incubator.apache.org"
	<cloudstack-dev@incubator.apache.org>
Date: Thu, 7 Mar 2013 12:45:29 +0000
Subject: RE: [PROPOSAL] BVT for CloudStack checkins
Thread-Topic: [PROPOSAL] BVT for CloudStack checkins
Thread-Index: Ac4al38DOjYZtFmYS82wZzdZtChODgAYE+FgAA5sCeA=
Message-ID: 
 <4B2EA7702410EE4FB4AF54931A777FED011EC67561BA@LONPMAILBOX01.citrite.net>
References: <20130306122142.GA12302@cloud-2.local>
 <CD5CC808.35E28%kelven.yang@citrix.com>
 <2529883E7B666F4E8F21F85AADA43CA7010C93E0F169@BANPMAILBOX01.citrite.net>
In-Reply-To: 
 <2529883E7B666F4E8F21F85AADA43CA7010C93E0F169@BANPMAILBOX01.citrite.net>
Accept-Language: en-US
Content-Language: en-US
acceptlanguage: en-US
Content-Type: text/plain; charset="us-ascii"
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0

+1 for a staging area - a master branch that passes BVT would be a big impr=
ovement.

-----Original Message-----
From: Koushik Das [mailto:koushik.das@citrix.com]=20
Sent: 07 March 2013 06:05
To: cloudstack-dev@incubator.apache.org
Subject: RE: [PROPOSAL] BVT for CloudStack checkins

+1 to the idea of staging area.
Each commit should be put in the staging area and on successful build and B=
VT run, it would get pushed to master. The staging area should maintain a q=
ueue of all the commits and process them in a sequential manner.
If it is possible to set up all these with the existing infra that would be=
 great.

> -----Original Message-----
> From: Kelven Yang [mailto:kelven.yang@citrix.com]
> Sent: Wednesday, March 06, 2013 11:52 PM
> To: cloudstack-dev@incubator.apache.org
> Subject: Re: [PROPOSAL] BVT for CloudStack checkins
>=20
> First +1 on BVT.
>=20
> Second, should we consider the idea of having a staging area for=20
> people to check-in? Which is that making master always the=20
> stable(reasonable) branch for main development, but whenever people=20
> make check-ins, it goes into staging first, and we have=20
> maintainers(could be automatic) to run whatever test framework we have=20
> and only perform automatic merge to master from staging area after a succ=
essful test-run?
>=20
> Kelven
>=20
> On 3/6/13 4:21 AM, "Prasanna Santhanam" <tsp@apache.org> wrote:
>=20
> >Great to see you kick this off Alex! I have a few ideas for this and=20
> >had been looking for help as well ..
> >
> >I had a draft lying around of an email you sent me long ago about=20
> >tiered testing and I think your proposal fits very well on this:
> >
> >The tl;dr of that conversation was as below
> >
> >- improve simulator for runtime testability
> >- customize to model and inject failures
> >- make a habit of writing tests around bug reports (I started tagging=20
> >tests since api_refactoring on JIRA already.
> >look for the integration-test label)
> >- make integration testing easier using factories and DSLs (from
> >  Chiradeep)
> >(part 1 of this work was started on this in the marvin-refactor
> >branch)
> >
> >More comments inline
> >
> >On Wed, Mar 06, 2013 at 12:12:11AM +0530, Alex Huang wrote:
> >> Hi All,
> >>
> >> As most of you are aware, the master branch keeps getting broken by=20
> >> checkins for various reasons.  Committers need to be more=20
> >> responsible about their checkins but I don't think we can depend on=20
> >> that happening.  There are various reasons.  The most obvious to me=20
> >> is that granting committership is not based on code competency.
> >> (And I don't think it should.)  Given that we need to build a BVT=20
> >> system for ensure that checkins do not break the branch.
> >>
> >> Here's my proposal:
> >>
> >> Existing components that we'll use.
> >>
> >> -          Citrix has contributed its testing to Apache.
> >>
> >> -          Apache CloudStack has already a simulator that's been used
> >>for scale testing.
> >>
> >> -          Marvin
> >>
> >> -          DevCloud-kvm
> >>
> >> Work Proposal:
> >>
> >> -          Convert the Citrix testing into three phases:
> >>
> >> o   Setup
> >>
> >> o   Test
> >>
> >> o   Verify
> >
> >I do the build, package, setup, test, verify in my integration test=20
> >pipeline and a similar pipeline for a developer combined into easily=20
> >runnable maven profiles/lifecycles/goals would be great to have.
> >
> >> -          Add a Setup and Verify phase for the simulator
> >> -          Add all of the agent commands necessary for the simulator t=
o
> >>pass the testing.
> >> -          Add a Setup and Verify phase for devCloud-kvm
> >
> >The setups exists as configs in the marvin sandbox already. Just need=20
> >to hook it up with mvn. For verify there is a simple python script=20
> >testSetupSuccess.py already that checks two things
> >
> >- system VMs are up
> >- built-in templates are downloaded
> >
> >This should be a good start IMO.
> >
> >Currently devcloud-kvm is a bit hard to run from a developer=20
> >environment. But it's great to have in a continuous environment=20
> >backed by a KVM host with a Linux 3.0 kernel. Marcus has written up=20
> >some pretty good documentation for this. If someone can help bring up=20
> >that setup I can assist in bringing up the tests using my devcloud-ci=20
> >scripts. I'm bringing up devcloud after Kelven 'alerted' us of the=20
> >memory fix.
> >
> >>
> >> -          Add two more profiles to pom
> >> o   Checkin-test-with-simulator: Runs the testing against the simulato=
r
> >> o   Checkin-test-with-devCloud: Runs the testing against devcloud
> >>
> >> -          All of the profiles will attempt to also check the merge
> >>list that Chip has proposed.'
> >
> >> -          We will also change marvin to easily add zones with
> >> actual hardware.  It will be based on a data driven document to do=20
> >> the setup.
> >This is currently partly doable using a properties file in the=20
> >sandbox $ python advanced_env.py -i xen.properties -o xen.cfg
> >
> >gives out a advanced zone config for the properties specified.
> >Is data driven similar or you are talking about a DSL that Edison=20
> >mentioned at CCC12?
> >
> >>
> >> For a developer to checkin:
> >>
> >> -          S/he must writes marvin tests for their feature and add
> >> it to the BVT.
> >I made some progress on this in my marvin-refactor branch and will=20
> >collect my thoughts in an FS that I am drafting. The marvin tests are=20
> >difficult to write in their current form. I'm following Chiradeep's=20
> >recommendation for providing factories and once those are baked a DSL=20
> >language devs can quickly mock down tests against a simulator using=20
> >the above mentioned mvn profiles.
> >
> >>
> >> -          S/he must run the checkin tests to verify everything works.
> >> -          If the feature contains a hardware/vpx component, simulatio=
n
> >>must be added.
> >>
> >> At this point, everything is about the developer writing in their=20
> >>feature branch and merging in.
> >>
> >> On infrastructure side:
> >>
> >> -          We'll setup continuous BVT based on the simulator.
> >I brought this up on the IRC and the list y'day, so +1 - happy to=20
> >help
> >>
> >> -          I again push that we must use Gerrit to test the code
> >> before it gets merge into the branch but I'll leave that for=20
> >> someone else to do that.
> >>
> >> Let me know what you guys think.  I'll probably break out a bvt=20
> >> branch to work on this.  Anyone want to join me?
> >
> >Count me in!
> >
> >--
> >Prasanna.,