cloudstack-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Erik Weber <terbol...@gmail.com>
Subject Re: [PROPOSAL] Using continuous integration to maintain our code quality...
Date Wed, 20 Aug 2014 19:24:50 GMT
On Wed, May 7, 2014 at 2:03 AM, Alex Huang <Alex.Huang@citrix.com> wrote:

> Hi All,
>
> This is something I brought up a long time ago but really didn't have the
> resources to get it all up and running until now.  Throughout the past
> year, Edison, Prasanna, Amogh, Bharat, Koushik, Talluri, and others have
> been chipping away at it.  At this point, we finally pull together a
> continuous integration setup that we can use to make sure that CloudStack
> master and the currently release branch are always stable.  This is getting
> pretty close to be completed and we like to share it with the community in
> hopes that we can reduce/eliminate that problems we've seen with our recent
> releases.  Currently, the physical hardware are hosted by Citrix but we'll
> be more than willing to donate the work to infra when that's all settled.
>
> This does require effort from the community to make a change in their
> development process.  These steps are detailed at [1].  I like to get
> feedback on what everyone think about this.
>
> What have we done:
>   - We replaced a large selection of the BVT tests to run with the
> simulator instead of actual hardware.  This shortens the duration of each
> BVT run.  Today, a BVT that runs tests for XenServer and KVM completes in
> 30-40 minutes.
>

How much is running with Simulator instead of actual hardware? My issue
with this is that you're testing against a flawless simulator in terms of
testing, while with actual hardware you are bound to hit bugs/issues that
might not be due to ACS code but ACS still has to handle it.

As an example, could you run a test on the tags '4.4.0' and '4.3.0' and
report the result? They both had fundamental flaws, where the one was
practically useless for a week or so, and the other had major issues with
KVM, and if the BVT doesn't encounter those because it's using the
simulator I see it as a burden rather than a gift, since you're relying on
a false result.

-- 
Erik

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message