cloudstack-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Stephen Turner <Stephen.Tur...@citrix.com>
Subject RE: Disabling failed test cases (was RE: Review Request 23605: CLOUDSTACK-7107: Disabling failed test cases)
Date Mon, 21 Jul 2014 09:07:20 GMT
In the case that it's a product bug, wouldn't it be better to keep running the test even if
you know it's going to fail? That way, you get a consistent view of the overall pass rate
from build to build. If you disable all the tests that are failing, you're going to get a
100% pass rate, but you can't see whether your quality is going up or down.

-- 
Stephen Turner


-----Original Message-----
From: Gaurav Aradhye [mailto:noreply@reviews.apache.org] On Behalf Of Gaurav Aradhye
Sent: 21 July 2014 09:58
To: Girish Shilamkar
Cc: Gaurav Aradhye; Hugo Trippaers; cloudstack
Subject: Re: Review Request 23605: CLOUDSTACK-7107: Disabling failed test cases



> On July 21, 2014, 1:03 p.m., Hugo Trippaers wrote:
> > Why would we want to disable test cases that fail? Doesn't this mean we need to
fix something else so they don't fail anymore?

Hi Hugo,

Whenever we found a test case failing, we create bug for that, may it be a test script issue
or product bug, so that the test case gets associated with a particular bug and it's easy
to track in future why it is failing.

Addition of this decorator BugId to test case skips the test in the run.

Whenever the bug gets fixed, then the person who has fixed the bug removes the BugId decorator
from test case so that the test case gets picked up in the next run.


- Gaurav


-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/23605/#review48204
-----------------------------------------------------------


On July 17, 2014, 1:17 p.m., Gaurav Aradhye wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/23605/
> -----------------------------------------------------------
> 
> (Updated July 17, 2014, 1:17 p.m.)
> 
> 
> Review request for cloudstack and Girish Shilamkar.
> 
> 
> Bugs: CLOUDSTACK-7074 and CLOUDSTACK-7107
>     https://issues.apache.org/jira/browse/CLOUDSTACK-7074
>     https://issues.apache.org/jira/browse/CLOUDSTACK-7107
> 
> 
> Repository: cloudstack-git
> 
> 
> Description
> -------
> 
> Disabling failed test cases on master.
> 
> 
> Diffs
> -----
> 
>   test/integration/smoke/test_primary_storage.py 66aec59 
>   test/integration/smoke/test_vm_life_cycle.py 240ab68 
> 
> Diff: https://reviews.apache.org/r/23605/diff/
> 
> 
> Testing
> -------
> 
> 
> Thanks,
> 
> Gaurav Aradhye
> 
>

Mime
View raw message