cloudstack-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Paul Angus <>
Subject RE: Guidelines for test developers
Date Wed, 18 Mar 2015 12:23:36 GMT
Thank you Gaurav,

It would great if the community developers could please express their support for these guidelines
(or start a conversation if you feel differently).

This way, this initiative will get traction.

@Gaurav, are these guidelines in the developer section of the CloudStack wiki?


Paul Angus
Cloud Architect
S: +44 20 3603 0540 | M: +447711418784 | T: CloudyAngus

-----Original Message-----
From: Gaurav Aradhye []
Sent: 18 March 2015 07:16
Subject: Guidelines for test developers

Hello all,

Last year after improving Marvin framework, we have been continuously spending time in improving
old test cases which are written in old style or they don't abide to certain guidelines, also
which don't use new functions available in marvin. Many times a test developer who is adding
test case for the first time or feature developers adding Basic Validation Tests
(BVTs) tend to copy paste the code available in certain test case and try to modify it according
to feature and commit it. This adds to inconsistencies.

As and when I touch a file for fixing an issue or adding/editing it, I try to incorporate
below guidelines and improve the existing code. But still few test files are not up to the
mark. Our final goal is to have good code in every file.So writing this mail to consolidate
few rules that should be known/considered by everyone adding tests to Marvin. Also, if you
touch a file, feel free to remove any inconsistencies that are already present in the file.

*1. Import * should always be avoided*. When I started two years back, and tried to understand
the code flow/ test cases, I could not easily understand from where the particular module
is imported. The imports must be specific.

When the imports are specific, it eliminates the possibility of test case failure due to invalid
import when specific import is removed from dependent module.

E.g. If your test case has following import.

from A import *

And it uses time module which is not imported explicitly in test case and comes from module
A. Then the test case will start failing when "import time"is removed from module A. You certainly
don't want this to happen.

*2. Maintaining Pep8 standards for python code.*

The code is read more often that it is written. Pep8 standards improve the readability of
the code making it consistent in style. There is a tool named "*autopep8*" which you can install
with pip install and then you can run following command on your test file.

autopep8 -i -a -a

This will make the file pep8 consistent and will also remove the white spaces. But some issues
need human intervention and can't be fixed with tool. For fixing those, check the pep8 issues
with "pep8" and fix manually.

*3. Keep only imports which are used* in the test suite and remove unwanted imports.

*4. Keep all the configuration parameters* (such as data which is passed to API while creating
Network offering, Service offering, account etc...) *in tools/marvin/marvin/config/
file*. Don't include them in test suite itself.

Many of the dictionaries are reusable and if you are adding a new test, there are only a few
dictionaries you will have to add in the file.

If any of the data contains URLs or any data which should be changed according to setup/env,
then include the dict in "*configurableData*"
section in file. This makes it easier to identify which data needs to be dynamic
according to the setup and which data doesn't need to be touched when env is changed.

*5. Before committing a test case, run it* with the latest branch against which you are adding
the test case and attach the results in Pull Request.
If in case change is very small, and doesn't need to be run, then at least check syntactical
errors with python command and also with the help of tools such as pyflakes.

6. If you add a new function in your test case and you think it can be used in future by other
test cases, then please *add that function to common or utils file* in Marvin. Don't keep
it local to test case. This will prevent multiple contributors adding same functions in their
test case to achieve a particular goal.

*7. Please make sure all the resources created through the test cases are deleted *when test
case execution completes, or even when the test case fails.

8. If same test case is to be run with different configuration or setting, you can *make use
of ddt library*. For example, if you have added test case for isolated networks, and you need
to run the same code for shared and VPC networks, then you don't need to add 3 test cases.
Just add relevant tags to the test case and you are good to go. Although you will need to
write code for handling those tags. It is already used in few test cases. A simple grep over
component folder and you can see how it is used.

This blog explains how it works.

I will check if this is in any wiki page currently, and edit it. Or will add a new page.
I hope everyone adding test cases follows above guidelines. Feel free to add more.

Find out more about ShapeBlue and our range of CloudStack related services

IaaS Cloud Design & Build<>
CSForge – rapid IaaS deployment framework<>
CloudStack Consulting<>
CloudStack Software Engineering<>
CloudStack Infrastructure Support<>
CloudStack Bootcamp Training Courses<>

This email and any attachments to it may be confidential and are intended solely for the use
of the individual to whom it is addressed. Any views or opinions expressed are solely those
of the author and do not necessarily represent those of Shape Blue Ltd or related companies.
If you are not the intended recipient of this email, you must neither take any action based
upon its contents, nor copy or show it to anyone. Please contact the sender if you believe
you have received this email in error. Shape Blue Ltd is a company incorporated in England
& Wales. ShapeBlue Services India LLP is a company incorporated in India and is operated
under license from Shape Blue Ltd. Shape Blue Brasil Consultoria Ltda is a company incorporated
in Brasil and is operated under license from Shape Blue Ltd. ShapeBlue SA Pty Ltd is a company
registered by The Republic of South Africa and is traded under license from Shape Blue Ltd.
ShapeBlue is a registered trademark.
View raw message