Mailing-List: contact dev-help@cloudstack.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@cloudstack.apache.org
From: Somesh Naidu <Somesh.Naidu@citrix.com>
To: "dev@cloudstack.apache.org" <dev@cloudstack.apache.org>
Subject: RE: [Blocker] test duplicated in test_scale_vm.py
Thread-Topic: [Blocker] test duplicated in test_scale_vm.py
Thread-Index: 
 AdD7SxHoeBvKd81NQHqmuL/NnZh2lAAAvg8QAAl14wAAAXKXgAAAFQMAAABcrwAAAGXtAAAAyxqAAADtYAAAAIn8AAACaQDg
Date: Wed, 30 Sep 2015 15:09:55 +0000
Message-ID: <7f946b6cc9b14ba49eb6517c7fd45b1d@FTLPEX02CL06.citrite.net>
References: 
 <93238155381A8343BD3FFE2CB75BC9E723B75EA8@SINPEX01CL01.citrite.net>
 <DEB2697CC649A142980A3B54B3B818C357359044@SINPEX01CL02.citrite.net>
 <4963CF2F-996E-4A03-9387-979BACC18A85@gmail.com>
 <93238155381A8343BD3FFE2CB75BC9E723B75FE0@SINPEX01CL01.citrite.net>
 <87357831-4FFB-41FA-977D-842819FE1725@gmail.com>
 <93238155381A8343BD3FFE2CB75BC9E723B76049@SINPEX01CL01.citrite.net>
 <506529B2-5703-4200-964F-1580642840F7@gmail.com>
 <93238155381A8343BD3FFE2CB75BC9E723B76121@SINPEX01CL01.citrite.net>
 <808E3CD0-10FB-49DD-B5DC-BEBCDE7DC87F@citrix.com>
 <CAGQtxvbUTgq5EtBYC08uCm2jn8KntN8Lwv0EW-cTKRLagVManA@mail.gmail.com>
In-Reply-To: 
 <CAGQtxvbUTgq5EtBYC08uCm2jn8KntN8Lwv0EW-cTKRLagVManA@mail.gmail.com>
Reply-To: Somesh Naidu <Somesh.Naidu@citrix.com>
Accept-Language: en-US
Content-Language: en-US
Content-Type: text/plain; charset="us-ascii"
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0

> that said we should still work towards a test set that can function as th=
e

So essentially, we are saying that the existing BVT isn't really a BVT and =
that we have to come up with it. From a user/customer standpoint this doesn=
't inspire confidence.

Anyway, is anybody willing to do this, as in, come up with a BVT/smoke/sani=
ty? Or do we have to rely on Citrix to do this?

Daan - do we have a tracker for this request?

Regards,
Somesh

-----Original Message-----
From: Daan Hoogland [mailto:daan.hoogland@gmail.com]=20
Sent: Wednesday, September 30, 2015 5:44 AM
To: dev
Subject: Re: [Blocker] test duplicated in test_scale_vm.py

I disagree, and let me carefully try to formulate why: BVT is useful and
'should' be run before any thing is declared good. The problem is the
ability for the community at large to run it and to fix it for any kind of
set-up. It should not be automatically treated as blockers but only if a
full analysis points to a root cause that can be considered a blocker.

that said we should still work towards a test set that can function as the
you describe the BVT and be automatically adding blockers.

regards,

On Wed, Sep 30, 2015 at 11:28 AM, Rajani Karuturi <
Rajani.Karuturi@citrix.com> wrote:

> I agree with Raja. BVT test case failures should be treated as blockers.
> Once the test issue is fixed, they might uncover new issues/regressions.
>
> ~Rajani
>
>
>
> On 30-Sep-2015, at 2:32 pm, Raja Pullela <raja.pullela@citrix.com> wrote:
>
> > The problem needs to be fixed - "have tested a solution that was
> discussed with koushik" - we will have to create PR with that change.  On=
ce
> the PR is pushed - we will have one less blocker/test failure.
> >
> > I agree this is not a cloudstack core code problem - but we should NOT
> make that a criteria for priority of a bug - just my take.  We should
> include BVT failures in the Blocker criteria (whether the fix is in the
> core functionality/test script).  Otherwise, we will not hit 100% passrat=
e
> on BVTs.
> >
> > You are right - only a subset of these tests are being run on "travis"
> otherwise we will catch more or some of these failures.
> >
> > -----Original Message-----
> > From: Sebastien Goasguen [mailto:runseb@gmail.com]
> > Sent: Wednesday, September 30, 2015 2:09 PM
> > To: dev@cloudstack.apache.org
> > Subject: Re: [Blocker] test duplicated in test_scale_vm.py
> >
> >
> >> On Sep 30, 2015, at 10:28 AM, Raja Pullela <raja.pullela@citrix.com>
> wrote:
> >>
> >> Yup... btw, I am on top of this and goal is to get to 100% passrate on
> BVTs!
> >>
> >
> > Yes I got it, and we all have the same goal.
> >
> > It does appear that not all smoke tests are run by Travis, otherwise we
> would catch any errors on simulator for every PR.
> >
> > My point is that if a failure in your BVT is due to the test itself it
> is not a cloudstack failure per se and hence not a blocker. Which all sai=
d
> and done is not a problem because it should be fixed now if I am
> understanding your assessment and Koushik report of this failure correctl=
y.
> >
> >
> >
> >> -----Original Message-----
> >> From: Sebastien Goasguen [mailto:runseb@gmail.com]
> >> Sent: Wednesday, September 30, 2015 1:48 PM
> >> To: dev@cloudstack.apache.org
> >> Subject: Re: [Blocker] test duplicated in test_scale_vm.py
> >>
> >>
> >>> On Sep 30, 2015, at 10:15 AM, Raja Pullela <raja.pullela@citrix.com>
> wrote:
> >>>
> >>> Sebastien, any BVT tests failing is a blocker and needs to be fixed -
> which is the reason for Blocker.  We already have a fix.
> >>>
> >>
> >> Right, but that's a fix in the test itself, no an issue with the
> cloudstack "core".
> >>
> >> In any case, great, one less blocker.
> >>
> >>> Koushik,
> >>>
> >>> modified the test "test_01_scale_vm" with required_hardware=3Dfalse o=
n
> simulator setup and it works.  So, I think we can let the second method
> go.  I will also test this on XS.
> >>> Raja
> >>>
> >>>
> >>> root@localhost:~/cloudstack# nosetests --with-marvin
> >>> --marvin-config=3D/root/cloudstack/setup/dev/local1.cfg
> >>> --zone=3DSandbox-simulator --hypervisor=3Dsimulator -a
> >>> tags=3Dbasic,required_hardware=3Dfalse
> >>> /root/cloudstack/test/integration/smoke/test_scale_vm.py
> >>>
> >>> =3D=3D=3D=3D Marvin Init Started =3D=3D=3D=3D
> >>>
> >>> =3D=3D=3D Marvin Parse Config Successful =3D=3D=3D
> >>>
> >>> =3D=3D=3D Marvin Setting TestData Successful=3D=3D=3D
> >>>
> >>> =3D=3D=3D=3D Log Folder Path: /tmp//MarvinLogs//Sep_30_2015_08_06_32_=
AWNF1O.
> >>> All logs will be available here =3D=3D=3D=3D
> >>>
> >>> =3D=3D=3D Marvin Init Logging Successful=3D=3D=3D
> >>>
> >>> =3D=3D=3D=3D Marvin Init Successful =3D=3D=3D=3D
> >>> =3D=3D=3Dfinal results are now copied to:
> >>> /tmp//MarvinLogs/test_scale_vm_OPS7AD=3D=3D=3D
> >>> root@localhost:~/cloudstack# cd /tmp//MarvinLogs/test_scale_vm_OPS7AD
> >>> root@localhost:/tmp/MarvinLogs/test_scale_vm_OPS7AD# ls
> >>> failed_plus_exceptions.txt  results.txt  runinfo.txt
> >>> root@localhost:/tmp/MarvinLogs/test_scale_vm_OPS7AD# vi  results.txt
> >>> root@localhost:/tmp/MarvinLogs/test_scale_vm_OPS7AD# ls -al total 48
> >>> drwxr-xr-x 2 root root  4096 Sep 30 08:07 .
> >>> drwxr-xr-x 8 root root  4096 Sep 30 08:06 ..
> >>> -rw-r--r-- 1 root root     0 Sep 30 08:06 failed_plus_exceptions.txt
> >>> -rw-r--r-- 1 root root   186 Sep 30 08:06 results.txt
> >>> -rw-r--r-- 1 root root 36164 Sep 30 08:06 runinfo.txt
> >>> root@localhost:/tmp/MarvinLogs/test_scale_vm_OPS7AD#
> >>> root@localhost:/tmp/MarvinLogs/test_scale_vm_OPS7AD# cat results.txt
> >>> Test scale virtual machine ... =3D=3D=3D TestName: test_01_scale_vm |
> >>> Status
> >>> : SUCCESS =3D=3D=3D ok
> >>>
> >>> ---------------------------------------------------------------------
> >>> -
> >>> Ran 1 test in 23.455s
> >>>
> >>> OK
> >>>
> >>> -----Original Message-----
> >>> From: Sebastien Goasguen [mailto:runseb@gmail.com]
> >>> Sent: Wednesday, September 30, 2015 1:04 PM
> >>> To: dev@cloudstack.apache.org
> >>> Subject: Re: [Blocker] test duplicated in test_scale_vm.py
> >>>
> >>>
> >>>> On Sep 30, 2015, at 9:10 AM, Koushik Das <koushik.das@citrix.com>
> wrote:
> >>>>
> >>>> Raja,
> >>>> The earlier scale vm test couldn't run on simulator because of the
> validation steps (IIRC it tries to connect to the real VM created during
> test). Since it wasn't possible to run with simulator, the new test was
> added to run just on the simulator. Check the required_hardware tag.
> >>>>
> >>>
> >>> Ok so that's not really a blocker.
> >>>
> >>> Can we reduce the severity to major ?
> >>> Or send a PR that will add this to the Travis run and run the correct
> test for simulator ?
> >>>
> >>> Or make you modifications in your BVT settings, re-run and close the
> issue ?
> >>>
> >>>
> >>>
> >>>> -----Original Message-----
> >>>> From: Raja Pullela [mailto:raja.pullela@citrix.com]
> >>>> Sent: Wednesday, 30 September 2015 12:12
> >>>> To: CloudStack Dev (dev@cloudstack.apache.org)
> >>>> Subject: [Blocker] test duplicated in test_scale_vm.py
> >>>>
> >>>> Hi,
> >>>>
> >>>> Filed this blocker on test case "test_scale_vm.py"  -
> >>>> https://issues.apache.org/jira/browse/CLOUDSTACK-8924
> >>>> Fixing this will bring Simulator Basic and XS Basic to 100% passrate=
.
> >>>>
> >>>> Best,
> >>>> Raja
> >>>>
> >>>
> >>
> >
>
>


--=20
Daan