cloudstack-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Edison Su <Edison...@citrix.com>
Subject RE: [PROPOSAL] BVT for CloudStack checkins
Date Tue, 05 Mar 2013 19:14:22 GMT
Recently, I read a topic about, "why automated test keeping failing"(http://programmers.stackexchange.com/questions/185819/why-does-automated-testing-keep-failing-in-my-company/185839#185839),
the comment is really resonated with me:

"Most unit tests are questionable in value. Since the vast majority of tests seem to be too
simple."
"It is much harder to write good testable code than just working code."
"Many types of designs/applications are better tested by automating tests at the module/package
level. In my experience, most coding errors are not because the code in a class was coded
incorrectly but because the coder didn't understand how their class was supposed to work with
other classes."

We'd better put our limited resource on BVT test on both simulator and real hardware, at least,
our BVT should cover all the hypervisors and all the cloudstack APIs. 

So +1.

> -----Original Message-----
> From: Alex Huang [mailto:Alex.Huang@citrix.com]
> Sent: Tuesday, March 05, 2013 10:42 AM
> To: cloudstack-dev@incubator.apache.org
> Subject: [PROPOSAL] BVT for CloudStack checkins
> 
> Hi All,
> 
> As most of you are aware, the master branch keeps getting broken by
> checkins for various reasons.  Committers need to be more responsible
> about their checkins but I don't think we can depend on that happening.
> There are various reasons.  The most obvious to me is that granting
> committership is not based on code competency.  (And I don't think it
> should.)  Given that we need to build a BVT system for ensure that checkins
> do not break the branch.
> 
> Here's my proposal:
> 
> Existing components that we'll use.
> 
> -          Citrix has contributed its testing to Apache.
> 
> -          Apache CloudStack has already a simulator that's been used for scale
> testing.
> 
> -          Marvin
> 
> -          DevCloud-kvm
> 
> Work Proposal:
> 
> -          Convert the Citrix testing into three phases:
> 
> o   Setup
> 
> o   Test
> 
> o   Verify
> 
> -          Add a Setup and Verify phase for the simulator
> 
> -          Add all of the agent commands necessary for the simulator to pass the
> testing.
> 
> -          Add a Setup and Verify phase for devCloud-kvm
> 
> -          Add two more profiles to pom
> 
> o   Checkin-test-with-simulator: Runs the testing against the simulator
> 
> o   Checkin-test-with-devCloud: Runs the testing against devcloud
> 
> -          All of the profiles will attempt to also check the merge list that Chip has
> proposed.'
> 
> -          We will also change marvin to easily add zones with actual hardware.  It
> will be based on a data driven document to do the setup.
> 
> For a developer to checkin:
> 
> -          S/he must writes marvin tests for their feature and add it to the BVT.
> 
> -          S/he must run the checkin tests to verify everything works.
> 
> -          If the feature contains a hardware/vpx component, simulation must be
> added.
> 
> At this point, everything is about the developer writing in their feature
> branch and merging in.
> 
> On infrastructure side:
> 
> -          We'll setup continuous BVT based on the simulator.
> 
> -          I again push that we must use Gerrit to test the code before it gets
> merge into the branch but I'll leave that for someone else to do that.
> 
> Let me know what you guys think.  I'll probably break out a bvt branch to
> work on this.  Anyone want to join me?
> 
> --Alex


Mime
View raw message