cloudstack-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Rajani Karuturi <Rajani.Karut...@citrix.com>
Subject Re: [Blocker] test duplicated in test_scale_vm.py
Date Wed, 30 Sep 2015 09:28:42 GMT
I agree with Raja. BVT test case failures should be treated as blockers. Once the test issue
is fixed, they might uncover new issues/regressions.

~Rajani



On 30-Sep-2015, at 2:32 pm, Raja Pullela <raja.pullela@citrix.com> wrote:

> The problem needs to be fixed - "have tested a solution that was discussed with koushik"
- we will have to create PR with that change.  Once the PR is pushed - we will have one less
blocker/test failure.  
> 
> I agree this is not a cloudstack core code problem - but we should NOT make that a criteria
for priority of a bug - just my take.  We should include BVT failures in the Blocker criteria
(whether the fix is in the core functionality/test script).  Otherwise, we will not hit 100%
passrate on BVTs. 
> 
> You are right - only a subset of these tests are being run on "travis" otherwise we will
catch more or some of these failures.  
> 
> -----Original Message-----
> From: Sebastien Goasguen [mailto:runseb@gmail.com] 
> Sent: Wednesday, September 30, 2015 2:09 PM
> To: dev@cloudstack.apache.org
> Subject: Re: [Blocker] test duplicated in test_scale_vm.py
> 
> 
>> On Sep 30, 2015, at 10:28 AM, Raja Pullela <raja.pullela@citrix.com> wrote:
>> 
>> Yup... btw, I am on top of this and goal is to get to 100% passrate on BVTs!  
>> 
> 
> Yes I got it, and we all have the same goal.
> 
> It does appear that not all smoke tests are run by Travis, otherwise we would catch any
errors on simulator for every PR.
> 
> My point is that if a failure in your BVT is due to the test itself it is not a cloudstack
failure per se and hence not a blocker. Which all said and done is not a problem because it
should be fixed now if I am understanding your assessment and Koushik report of this failure
correctly.
> 
> 
> 
>> -----Original Message-----
>> From: Sebastien Goasguen [mailto:runseb@gmail.com]
>> Sent: Wednesday, September 30, 2015 1:48 PM
>> To: dev@cloudstack.apache.org
>> Subject: Re: [Blocker] test duplicated in test_scale_vm.py
>> 
>> 
>>> On Sep 30, 2015, at 10:15 AM, Raja Pullela <raja.pullela@citrix.com> wrote:
>>> 
>>> Sebastien, any BVT tests failing is a blocker and needs to be fixed - which is
the reason for Blocker.  We already have a fix.   
>>> 
>> 
>> Right, but that’s a fix in the test itself, no an issue with the cloudstack “core”.
>> 
>> In any case, great, one less blocker. 
>> 
>>> Koushik,
>>> 
>>> modified the test "test_01_scale_vm" with required_hardware=false on simulator
setup and it works.  So, I think we can let the second method go.  I will also test this on
XS.
>>> Raja
>>> 
>>> 
>>> root@localhost:~/cloudstack# nosetests --with-marvin 
>>> --marvin-config=/root/cloudstack/setup/dev/local1.cfg
>>> --zone=Sandbox-simulator --hypervisor=simulator -a 
>>> tags=basic,required_hardware=false
>>> /root/cloudstack/test/integration/smoke/test_scale_vm.py
>>> 
>>> ==== Marvin Init Started ====
>>> 
>>> === Marvin Parse Config Successful ===
>>> 
>>> === Marvin Setting TestData Successful===
>>> 
>>> ==== Log Folder Path: /tmp//MarvinLogs//Sep_30_2015_08_06_32_AWNF1O. 
>>> All logs will be available here ====
>>> 
>>> === Marvin Init Logging Successful===
>>> 
>>> ==== Marvin Init Successful ====
>>> ===final results are now copied to: 
>>> /tmp//MarvinLogs/test_scale_vm_OPS7AD===
>>> root@localhost:~/cloudstack# cd /tmp//MarvinLogs/test_scale_vm_OPS7AD
>>> root@localhost:/tmp/MarvinLogs/test_scale_vm_OPS7AD# ls 
>>> failed_plus_exceptions.txt  results.txt  runinfo.txt 
>>> root@localhost:/tmp/MarvinLogs/test_scale_vm_OPS7AD# vi  results.txt 
>>> root@localhost:/tmp/MarvinLogs/test_scale_vm_OPS7AD# ls -al total 48 
>>> drwxr-xr-x 2 root root  4096 Sep 30 08:07 .
>>> drwxr-xr-x 8 root root  4096 Sep 30 08:06 ..
>>> -rw-r--r-- 1 root root     0 Sep 30 08:06 failed_plus_exceptions.txt
>>> -rw-r--r-- 1 root root   186 Sep 30 08:06 results.txt
>>> -rw-r--r-- 1 root root 36164 Sep 30 08:06 runinfo.txt 
>>> root@localhost:/tmp/MarvinLogs/test_scale_vm_OPS7AD#
>>> root@localhost:/tmp/MarvinLogs/test_scale_vm_OPS7AD# cat results.txt 
>>> Test scale virtual machine ... === TestName: test_01_scale_vm | 
>>> Status
>>> : SUCCESS === ok
>>> 
>>> ---------------------------------------------------------------------
>>> -
>>> Ran 1 test in 23.455s
>>> 
>>> OK
>>> 
>>> -----Original Message-----
>>> From: Sebastien Goasguen [mailto:runseb@gmail.com]
>>> Sent: Wednesday, September 30, 2015 1:04 PM
>>> To: dev@cloudstack.apache.org
>>> Subject: Re: [Blocker] test duplicated in test_scale_vm.py
>>> 
>>> 
>>>> On Sep 30, 2015, at 9:10 AM, Koushik Das <koushik.das@citrix.com> wrote:
>>>> 
>>>> Raja,
>>>> The earlier scale vm test couldn't run on simulator because of the validation
steps (IIRC it tries to connect to the real VM created during test). Since it wasn't possible
to run with simulator, the new test was added to run just on the simulator. Check the required_hardware
tag.
>>>> 
>>> 
>>> Ok so that’s not really a blocker. 
>>> 
>>> Can we reduce the severity to major ?
>>> Or send a PR that will add this to the Travis run and run the correct test for
simulator ?
>>> 
>>> Or make you modifications in your BVT settings, re-run and close the issue ?
>>> 
>>> 
>>> 
>>>> -----Original Message-----
>>>> From: Raja Pullela [mailto:raja.pullela@citrix.com]
>>>> Sent: Wednesday, 30 September 2015 12:12
>>>> To: CloudStack Dev (dev@cloudstack.apache.org)
>>>> Subject: [Blocker] test duplicated in test_scale_vm.py
>>>> 
>>>> Hi,
>>>> 
>>>> Filed this blocker on test case "test_scale_vm.py"  -
>>>> https://issues.apache.org/jira/browse/CLOUDSTACK-8924
>>>> Fixing this will bring Simulator Basic and XS Basic to 100% passrate.
>>>> 
>>>> Best,
>>>> Raja
>>>> 
>>> 
>> 
> 


Mime
View raw message