incubator-cloudstack-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Mice Xia" <mice_...@tcloudcomputing.com>
Subject RE: [DISCUSS] Concerns on testing and improvements
Date Thu, 31 Jan 2013 05:23:20 GMT
Prasanna:

2) More and more requests from contributors on how to write unit-tests show up on our lists
every week.

I agree we need more unit tests. At this moment there isn’t much detailed guideline or material
about how to write a unit test with unified style.
For example, before 4.1.0, mocks are injected with MockComponentLocator and these mock objects
must be implemented manually. Later we bring mockito back and simplify the test codes. Now
with spring being introduced, should we also use spring test framework?
And for different layers, the way to implement unit tests might be different, to test a Resource
class, we need a lot of mocks/stubs of external library; to test a DAO, should we prepare
the database with test data?
I think we need some consensus on ' what should be unit tested' and 'the recommend way to
write a unit test'.

4. Easing test writing is my primary concern at the moment and I've been refactoring marvin
to accomadate a easier dsl style test composition. I hope to have a working test case sometime
next week.

I haven’t followed this for a while, and I don’t use devCloud . Are there any documents
describing how to run Marvin test with simulator/hypervisors?

Regards
Mice

-----Original Message-----
From: prasanna [mailto:srivatsav.prasanna@gmail.com] On Behalf Of Prasanna Santhanam
Sent: Wednesday, January 30, 2013 2:02 AM
To: cloudstack-dev@incubator.apache.org
Subject: [DISCUSS] Concerns on testing and improvements

On Tue, Jan 29, 2013 at 10:43:33AM -0500, Alex Huang wrote:
> > > It is also true CloudStack does not have a good automated 
> > > regression test suite to make sure a check-in like this does not 
> > > break some other features in CloudStack. But lack of a thorough 
> > > automated regression suite a problem with CloudStack in general. We've let
in other big changes in this release.
> > > I know developers who wrote this check-in have thoroughly 
> > > regression
> > tested
> > > to the best of their abilities.
> > >
> > >
> > This is  a point I tried to make to a few folks before CloudStack 
> > came here. If you want to provide a smooth path to committership you 
> > need a solid regression testing framework (with near total coverage) 
> > and an environment. Feeling blind and vulnerable to big changes like 
> > this limits trust and we can't have a lack of trust. If you want 
> > this project growing faster and healthier regression testing is key: it has a social
impact.
> > 
> 
> +1  Again Agree on this.  CloudStack has been lacking a good
> automated regression tests.  We need to start doing something about 
> it.

This warrants its own thread (which unfortunately has become a common feature on our lists)

IMO - cloudstack is and remains largely untestable! I've often seen these requests for more
tests spring (no pun intended) up when a large code merge is called for. Happened with api_refactoring
as well. But the truth of the matter is "tests are an afterthought". 

If you don't believe me:

1) One only needs look at the list of features waiting on ipclearance without much to call
in terms of tests. Feature is developed - testing can come later.

2) More and more requests from contributors on how to write unit-tests show up on our lists
every week.

3) Tests cannot magically appear unless there is a concerted effort in testing right from
feature conception which is lacking.

There's another fundamental problem. Traditionally cloudstack development has happened separate
from testing processes. Code is 'thrown-over-the-wall' to be tested which is not very different
from how things are today. With that attitude I only see test playing 'catch-up' to the almost
*maddening* pace of development.


And now for some solutions:

1. Start test development alongside feature development. Attach a test engineer if possible
to every feature branch. Make unit-test writing mandatory for both committers and non-committers.

2. I've been working with some qa folk within my company to address the issue more proactively.
They (being non-committers) look forward to start working on github [1] along with some of
our engineers who wrote the original integration tests. The test activity will proceed in
the same feature branch as being used by the developer. And will be connected to appropriate
infrastructure. But likely this will happen only after 4.1

3. Cloudstack needs massive infrastructure, build and packaging pieces to come together for
doing live provider testing. While I got this up for 4.0 with a live environment, this hasn't
received much love on master. Over the past couple days I've been testing the replicability
of this infrastructure and have been moving the pieces to builds.a.o

4. Easing test writing is my primary concern at the moment and I've been refactoring marvin
to accomadate a easier dsl style test composition. I hope to have a working test case sometime
next week.

Much of this has come too little, too late. However I beseech others to pitch in with ideas
and contributions on improving testability on the whole to ease this pain for the forthcoming
releases.

[1] https://github.com/cloudstack-qa

--
Prasanna.,
PS: Forgot to break the thread earlier in haste.
Mime
View raw message