Mailing-List: contact dev-help@cloudstack.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@cloudstack.apache.org
Received-SPF: pass (nike.apache.org: domain of LSimons@schubergphilis.com
 designates 195.66.90.57 as permitted sender)
From: Leo Simons <LSimons@schubergphilis.com>
To: "<dev@cloudstack.apache.org>" <dev@cloudstack.apache.org>
Subject: Re: Guidelines for test developers
Thread-Topic: Guidelines for test developers
Thread-Index: AQHQYUukhZ1gk65atkmvS7v/xgtbtZ0s62xj
Date: Wed, 25 Mar 2015 08:38:03 +0000
Message-ID: <E23E427A-323D-48C4-95F2-F15B9F8A03EF@schubergphilis.com>
References: 
 <CAFU5GBQpxuj=dZga-aU3wzdRoVRB7W3cU3Knr+OwdHSX1J3A5A@mail.gmail.com>
In-Reply-To: 
 <CAFU5GBQpxuj=dZga-aU3wzdRoVRB7W3cU3Knr+OwdHSX1J3A5A@mail.gmail.com>
Accept-Language: en-US
Content-Language: en-US
Content-Type: text/plain; charset="us-ascii"
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0

Thanks for writing this up, very useful!

Cheers,

Leo

> On 18 Mar 2015, at 08:17, Gaurav Aradhye <gaurav.aradhye@clogeny.com> wro=
te:
>=20
> Hello all,
>=20
> Last year after improving Marvin framework, we have been continuously
> spending time in improving old test cases which are written in old style =
or
> they don't abide to certain guidelines, also which don't use new function=
s
> available in marvin. Many times a test developer who is adding test case
> for the first time or feature developers adding Basic Validation Tests
> (BVTs) tend to copy paste the code available in certain test case and try
> to modify it according to feature and commit it. This adds to
> inconsistencies.
>=20
> As and when I touch a file for fixing an issue or adding/editing it, I tr=
y
> to incorporate below guidelines and improve the existing code. But still
> few test files are not up to the mark. Our final goal is to have good cod=
e
> in every file.So writing this mail to consolidate few rules that should b=
e
> known/considered by everyone adding tests to Marvin. Also, if you touch a
> file, feel free to remove any inconsistencies that are already present in
> the file.
>=20
> *1. Import * should always be avoided*. When I started two years back, an=
d
> tried to understand the code flow/ test cases, I could not easily
> understand from where the particular module is imported. The imports must
> be specific.
>=20
> When the imports are specific, it eliminates the possibility of test case
> failure due to invalid import when specific import is removed from
> dependent module.
>=20
> E.g. If your test case has following import.
>=20
> from A import *
>=20
> And it uses time module which is not imported explicitly in test case and
> comes from module A. Then the test case will start failing when "import
> time"is removed from module A. You certainly don't want this to happen.
>=20
> *2. Maintaining Pep8 standards for python code.*
>=20
> The code is read more often that it is written. Pep8 standards improve th=
e
> readability of the code making it consistent in style. There is a tool
> named "*autopep8*" which you can install with pip install and then you ca=
n
> run following command on your test file.
>=20
> autopep8 -i -a -a testFile.py
>=20
> This will make the file pep8 consistent and will also remove the white
> spaces. But some issues need human intervention and can't be fixed with
> tool. For fixing those, check the pep8 issues with "pep8 testFile.py" and
> fix manually.
>=20
> *3. Keep only imports which are used* in the test suite and remove unwant=
ed
> imports.
>=20
> *4. Keep all the configuration parameters* (such as data which is passed =
to
> API while creating Network offering, Service offering, account etc...) *i=
n
> tools/marvin/marvin/config/test_data.py file*. Don't include them in test
> suite itself.
>=20
> Many of the dictionaries are reusable and if you are adding a new test,
> there are only a few dictionaries you will have to add in the file.
>=20
> If any of the data contains URLs or any data which should be changed
> according to setup/env, then include the dict in "*configurableData*"
> section in test_data.py file. This makes it easier to identify which data
> needs to be dynamic according to the setup and which data doesn't need to
> be touched when env is changed.
>=20
> *5. Before committing a test case, run it* with the latest branch against
> which you are adding the test case and attach the results in Pull Request=
.
> If in case change is very small, and doesn't need to be run, then at leas=
t
> check syntactical errors with python command and also with the help of
> tools such as pyflakes.
>=20
> 6. If you add a new function in your test case and you think it can be us=
ed
> in future by other test cases, then please *add that function to common o=
r
> utils file* in Marvin. Don't keep it local to test case. This will preven=
t
> multiple contributors adding same functions in their test case to achieve=
 a
> particular goal.
>=20
> *7. Please make sure all the resources created through the test cases are
> deleted *when test case execution completes, or even when the test case
> fails.
>=20
> 8. If same test case is to be run with different configuration or setting=
,
> you can *make use of ddt library*. For example, if you have added test ca=
se
> for isolated networks, and you need to run the same code for shared and V=
PC
> networks, then you don't need to add 3 test cases. Just add relevant tags
> to the test case and you are good to go. Although you will need to write
> code for handling those tags. It is already used in few test cases. A
> simple grep over component folder and you can see how it is used.
>=20
> This blog explains how it works.
> https://technomilk.wordpress.com/2012/02/12/multiplying-python-unit-test-=
cases-with-different-sets-of-data/
>=20
> I will check if this is in any wiki page currently, and edit it. Or will
> add a new page.
> I hope everyone adding test cases follows above guidelines. Feel free to
> add more.
>=20
> Regards,
> Gaurav