incubator-cloudstack-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Edison Su <Edison...@citrix.com>
Subject RE: [DISCUSS] Concerns on testing and improvements
Date Thu, 31 Jan 2013 06:25:34 GMT


> -----Original Message-----
> From: Mice Xia [mailto:mice_xia@tcloudcomputing.com]
> Sent: Wednesday, January 30, 2013 9:23 PM
> To: cloudstack-dev@incubator.apache.org
> Subject: RE: [DISCUSS] Concerns on testing and improvements
> 
> Prasanna:
> 
> 2) More and more requests from contributors on how to write unit-tests
> show up on our lists every week.
> 
> I agree we need more unit tests. At this moment there isn’t much detailed
> guideline or material about how to write a unit test with unified style.
> For example, before 4.1.0, mocks are injected with MockComponentLocator
> and these mock objects must be implemented manually. Later we bring
> mockito back and simplify the test codes. Now with spring being introduced,
> should we also use spring test framework?


Yes, that's why we need to merge javelin into master ASAP, writing unit test with Spring and
Mockito and Junit/TestNG, is much easier.

> And for different layers, the way to implement unit tests might be different,
> to test a Resource class, we need a lot of mocks/stubs of external library; to
> test a DAO, should we prepare the database with test data?

> I think we need some consensus on ' what should be unit tested' and 'the
> recommend way to write a unit test'.
> 
> 4. Easing test writing is my primary concern at the moment and I've been
> refactoring marvin to accomadate a easier dsl style test composition. I hope
> to have a working test case sometime next week.
> 
> I haven’t followed this for a while, and I don’t use devCloud . Are there any
> documents describing how to run Marvin test with simulator/hypervisors?
> 
> Regards
> Mice
> 
> -----Original Message-----
> From: prasanna [mailto:srivatsav.prasanna@gmail.com] On Behalf Of
> Prasanna Santhanam
> Sent: Wednesday, January 30, 2013 2:02 AM
> To: cloudstack-dev@incubator.apache.org
> Subject: [DISCUSS] Concerns on testing and improvements
> 
> On Tue, Jan 29, 2013 at 10:43:33AM -0500, Alex Huang wrote:
> > > > It is also true CloudStack does not have a good automated
> > > > regression test suite to make sure a check-in like this does not
> > > > break some other features in CloudStack. But lack of a thorough
> > > > automated regression suite a problem with CloudStack in general.
> We've let in other big changes in this release.
> > > > I know developers who wrote this check-in have thoroughly
> > > > regression
> > > tested
> > > > to the best of their abilities.
> > > >
> > > >
> > > This is  a point I tried to make to a few folks before CloudStack
> > > came here. If you want to provide a smooth path to committership you
> > > need a solid regression testing framework (with near total coverage)
> > > and an environment. Feeling blind and vulnerable to big changes like
> > > this limits trust and we can't have a lack of trust. If you want
> > > this project growing faster and healthier regression testing is key: it has
a
> social impact.
> > >
> >
> > +1  Again Agree on this.  CloudStack has been lacking a good
> > automated regression tests.  We need to start doing something about
> > it.
> 
> This warrants its own thread (which unfortunately has become a common
> feature on our lists)
> 
> IMO - cloudstack is and remains largely untestable! I've often seen these
> requests for more tests spring (no pun intended) up when a large code
> merge is called for. Happened with api_refactoring as well. But the truth of
> the matter is "tests are an afterthought".
> 
> If you don't believe me:
> 
> 1) One only needs look at the list of features waiting on ipclearance without
> much to call in terms of tests. Feature is developed - testing can come later.
> 
> 2) More and more requests from contributors on how to write unit-tests
> show up on our lists every week.
> 
> 3) Tests cannot magically appear unless there is a concerted effort in testing
> right from feature conception which is lacking.
> 
> There's another fundamental problem. Traditionally cloudstack development
> has happened separate from testing processes. Code is 'thrown-over-the-
> wall' to be tested which is not very different from how things are today. With
> that attitude I only see test playing 'catch-up' to the almost *maddening*
> pace of development.
> 
> 
> And now for some solutions:
> 
> 1. Start test development alongside feature development. Attach a test
> engineer if possible to every feature branch. Make unit-test writing
> mandatory for both committers and non-committers.
> 
> 2. I've been working with some qa folk within my company to address the
> issue more proactively. They (being non-committers) look forward to start
> working on github [1] along with some of our engineers who wrote the
> original integration tests. The test activity will proceed in the same feature
> branch as being used by the developer. And will be connected to appropriate
> infrastructure. But likely this will happen only after 4.1
> 
> 3. Cloudstack needs massive infrastructure, build and packaging pieces to
> come together for doing live provider testing. While I got this up for 4.0 with
> a live environment, this hasn't received much love on master. Over the past
> couple days I've been testing the replicability of this infrastructure and have
> been moving the pieces to builds.a.o
> 
> 4. Easing test writing is my primary concern at the moment and I've been
> refactoring marvin to accomadate a easier dsl style test composition. I hope
> to have a working test case sometime next week.
> 
> Much of this has come too little, too late. However I beseech others to pitch
> in with ideas and contributions on improving testability on the whole to ease
> this pain for the forthcoming releases.
> 
> [1] https://github.com/cloudstack-qa
> 
> --
> Prasanna.,
> PS: Forgot to break the thread earlier in haste.
Mime
View raw message