Mailing-List: contact dev-help@cloudstack.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@cloudstack.apache.org
Received-SPF: pass (athena.apache.org: domain of Alex.Huang@citrix.com
 designates 66.165.176.63 as permitted sender)
From: Alex Huang <Alex.Huang@citrix.com>
To: "dev@cloudstack.apache.org" <dev@cloudstack.apache.org>
Subject: RE: [rant] stupid test cases
Thread-Topic: [rant] stupid test cases
Thread-Index: AQHOg3hcq9n47BJEA06b2HT+sd1UAJlqYs0AgAEQYgD//72ZUA==
Date: Fri, 19 Jul 2013 01:12:14 +0000
Message-ID: <20CF38CB4385CE4D9D1558D52A0FC05806ABBB@SJCPEX01CL03.citrite.net>
References: <20130718053238.GA5845@cloud-2.local>
 <CALFpzo64STN-T0pdd-wnBUR0Aba939N-UptghO5tumZTH6=A-g@mail.gmail.com>
 <83F77FF1BD50194AB65AFA5479D082A7065330@SJCPEX01CL02.citrite.net>
In-Reply-To: <83F77FF1BD50194AB65AFA5479D082A7065330@SJCPEX01CL02.citrite.net>
Accept-Language: en-US
Content-Language: en-US
Content-Type: text/plain; charset="us-ascii"
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0

I disagree.  Error handling should be part of our testing.

We should incorporate the simulator into the BVT and regression tests.  On =
testcases that really is to test the business logic rather than the provisi=
oning code, the test case should perform all of the provisioning on the sim=
ulator instead.  Then simulator can be programmed to simulate VM stopped fa=
ilure etc and see how the business responds to these problems.

--Alex

> -----Original Message-----
> From: Anthony Xu [mailto:Xuefei.Xu@citrix.com]
> Sent: Thursday, July 18, 2013 3:02 PM
> To: dev@cloudstack.apache.org
> Subject: RE: [rant] stupid test cases
>=20
> +1   VM can be in "Stopped" state
>=20
>=20
> Anthony
>=20
> -----Original Message-----
> From: Marcus Sorensen [mailto:shadowsor@gmail.com]
> Sent: Wednesday, July 17, 2013 10:47 PM
> To: dev@cloudstack.apache.org
> Subject: Re: [rant] stupid test cases
>=20
> I can understand that we may want to test that everything related to the
> domain gets cleaned up properly. We have run into all sorts of things whe=
n
> deleting accounts, for example where resources won't clean up because the
> account is gone and we throw null pointers because a bunch of code looks =
up
> account when deleting. However, to your point, VMs can be created in a
> "stopped" state and that wouldn't incur the overhead of deployment.
> On Jul 17, 2013 11:33 PM, "Prasanna Santhanam" <tsp@apache.org> wrote:
>=20
> > I was just going through one of the automated test cases and I find it
> > really silly that there's the following test:
> >
> > def test_forceDeleteDomain(self):
> >         """ Test delete domain force option"""
> >
> >         # Steps for validations
> >         # 1. create a domain DOM
> >         # 2. create 2 users under this domain
> >         # 3. deploy 1 VM into each of these user accounts
> >         # 4. create PF / FW rules for port 22 on these VMs for their
> >         #    respective accounts
> >         # 5. delete the domain with force=3Dtrue option
> >         # Validate the following
> >         # 1. listDomains should list the created domain
> >         # 2. listAccounts should list the created accounts
> >         # 3. listvirtualmachines should show the Running VMs
> >         # 4. PF and FW rules should be shown in listFirewallRules
> >         # 5. domain should delete successfully and above three list cal=
ls
> >         #    should show all the resources now deleted. listRouters sho=
uld
> >         #    not return any routers in the deleted accounts/domains
> >
> > Why would one need the overhead of creating VMs in a domain deletion
> test?
> > Do
> > we not understand that the basics accounts/domains/ etc that
> > cloudstack has nothing to do with the virtual machines? This kind of a
> > test slows down other useful tests that we could be running. More over
> > when this fails in the VM creation step I'd have to go in and analyse
> > logs to realize that deletedomain was perhaps fine but vm creation has
> > failed.
> >
> > That's a pointless effort. I'm sure there are others in the automated
> > tests that do this kind of wasteful testing. So please please pleaes
> > &*()#@()# please review test plans before automating them!
> >
> > I'm not going to be looking at this forever to fix these issues when
> > we want to see pretty metrics and numbers.
> >
> > --
> > Prasanna.,
> >
> > ------------------------
> > Powered by BigRock.com
> >
> >