Mailing-List: contact issues-help@cloudstack.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@cloudstack.apache.org
Received-SPF: pass (athena.apache.org: local policy includes SPF record at
 spf.trusted-forwarder.org)
Date: Mon, 28 Oct 2013 12:06:48 +0530
From: Prasanna Santhanam <tsp@apache.org>
To: dev@cloudstack.apache.org
Cc: "issues@cloudstack.apache.org" <issues@cloudstack.apache.org>
Subject: Re: [DISCUSS] (CLOUDSTACK-4971) Verifying Test Beds : prerequisites
 for launching tests
Message-ID: <20131028063648.GA3972@cloud-2.local>
Mail-Followup-To: dev@cloudstack.apache.org,
	"issues@cloudstack.apache.org" <issues@cloudstack.apache.org>
References: <JIRA.12676058.1382939335846@arcas>
 <JIRA.12676058.1382939335846.9754.1382939379493@arcas>
 <20131028061519.GA3424@cloud-2.local>
 <CB688D716959224C95995F52DE5E82420EBA07B4@SINPEX01CL02.citrite.net>
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
In-Reply-To: 
 <CB688D716959224C95995F52DE5E82420EBA07B4@SINPEX01CL02.citrite.net>
User-Agent: Mutt/1.5.21 (2010-09-15)

On Mon, Oct 28, 2013 at 06:24:05AM +0000, Santhosh Edukulla wrote:
> Here, we mean test setup, not marvin setup ( or setup.py ) . 

Sure - I understand that :). Marvin's setup is only a build+packaging
file. I should've been clear - when I say deployment scripts those are
the cloud-autodeploy scripts that setup the testbed. Marvin's dumb in
that respect and only comes in when cloudstack is configured and tests
are launched.

> 
> Each test has their own dependency, we are having
> setup,flow,teardown for test module. By below content of bug
> description, we mean verifying the test dependencies under setup
> class of a test feature or module not the marvin setup. The test
> feature or module setup can take care to verify these dependencies
> and exit with better log and reporting as against reporting the test
> cases as failures.

Right - so you're saying you'll change the tests. But as I mentioned
with 'netaddr' it's the problem of the python environment running the
test. The test is free to import 'netaddr' but not install it.

> 
> Some dependencies at marvin setup can be verified, but the below
> purpose is to ensure test dependencies are met before running tests.
> The goal is to make less or zero % test script errors, against
> genuine product bugs as failures. 
> 

Okay - by 'met before running' you mean during deployment or before
test launch? Or as part of the setUpClass? If it is part of the
setUpClass. Isn't that just a guideline to the test case writer?

> Santhosh
> ________________________________________
> From: Prasanna Santhanam [tsp@apache.org]
> Sent: Monday, October 28, 2013 2:15 AM
> To: dev@cloudstack.apache.org
> Cc: issues@cloudstack.apache.org
> Subject: [DISCUSS] (CLOUDSTACK-4971) Verifying Test Beds : prerequisites for launching tests
> 
> I like the concept of a health check before launching tests. But it's
> a difficult distinction to make whether to run a specific test because
> its required hardware component/external dependency is not
> available/not ready in the deployment OR have the test ensure that the
> deployment is suitable for its run.
> 
> So our current health check only ensures two things
> - SystemVMs are up and running
> - Default Templates are 'Ready' in the system
> 
> Take the example of LDAP here that you quoted. In such a case I'd
> prefer that the test itself check for the deployment guarantees
> without pushing that to the framework or the deployment scripts to
> decipher.
> 
> There is a counter case - some tests use the netaddr module to do
> subnet arithmetic. For these to run we need to ensure the virtualenv
> under which the test module runs contains netaddr. This is something
> that can be ensured by the deployment scripts.
> 
> So this is a question of guidelines to test authors on what to expect
> and what not to expect when writing a test. Also how to have the
> dependencies added automatically before the test gets executed. I
> don't see a one-size-fits-all solution.
> 
> On Mon, Oct 28, 2013 at 05:49:39AM +0000, Santhosh Kumar Edukulla (JIRA) wrote:
> > Santhosh Kumar Edukulla created CLOUDSTACK-4971:
> >
> > 1, Currently test scripts are running despite the test setup has
> > issues. EX: In one of the test scenario, where ldap server was down,
> > then test suite depending upon them was still running.
> >
> > 2, Ideally, the setup should check for these dependencies before
> > running the test script and exit wherever possible with proper log
> > and reporting .
> >
> > 3,  We may not be able to cover all setup issues, but the purpose of
> > setup is also to check for its existence and dependencies. If
> > scripts depending upon particular server conditions, should have a
> > proper setup verifying the server existence before proceeding
> > further.
> >
> > 4, I believe running test cases when setup is down is also not
> > idealistic.
> >
> > 5. Purpose is to reduce script errors vs failures due to product
> > bugs.
> >
> > 6. Logging this bug to track the setup issue cases or script
> > failures.
> >
> --
> Prasanna.,
> 
> ------------------------
> Powered by BigRock.com

-- 
Prasanna.,

------------------------
Powered by BigRock.com