incubator-cloudstack-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Marcus Sorensen <shadow...@gmail.com>
Subject Re: [DISCUSS] Concerns on testing and improvements
Date Thu, 31 Jan 2013 16:59:48 GMT
We do that, but the majority of what's useful to test is advanced zone
stuff. Which we can also do in devcloud with the right config. That's
sort of what I was getting at, that we should standardize on an
advanced config for devcloud so that everyone is writing against and
testing the same.

On Thu, Jan 31, 2013 at 12:29 AM, Ahmad Emneina <aemneina@gmail.com> wrote:
> Can we not group the tests vs standardizing all test to work against
> devcloud? that way one can run the devcloud test suite against devcloud and
> not be bothered with failing advanced zone tests.
>
>
> On Wed, Jan 30, 2013 at 11:03 PM, Marcus Sorensen <shadowsor@gmail.com>wrote:
>
>> On the integration testing, my impressions are that they're written
>> against a specific environment. That's probably expected, being an
>> actual functionality test, but it makes it hard to actually run them.
>> Some of them work against devcloud in a basic zone, others work for an
>> advanced setup, some tests don't seem to work at all and I'm not sure
>> if they're just unfinished or specific to someone's test environment.
>> Everyone might be writing tests but they only work in their
>> environment.
>>
>> I've mentioned this before, but it should be possible to standardize
>> on a marvin config that is self-contained (management server inside),
>> that anyone can download and run the tests on, write tests against.
>> Even possibly have a devcloud vm that pulls down master on a cron job,
>> runs the tests, and emails/reports the results every day. It should
>> even be possible to have a devcloud and devcloud-kvm with the same
>> configs (same address ranges, etc) so the tests can be run against
>> both.
>>
>> On Wed, Jan 30, 2013 at 11:25 PM, Edison Su <Edison.su@citrix.com> wrote:
>> >
>> >
>> >> -----Original Message-----
>> >> From: Mice Xia [mailto:mice_xia@tcloudcomputing.com]
>> >> Sent: Wednesday, January 30, 2013 9:23 PM
>> >> To: cloudstack-dev@incubator.apache.org
>> >> Subject: RE: [DISCUSS] Concerns on testing and improvements
>> >>
>> >> Prasanna:
>> >>
>> >> 2) More and more requests from contributors on how to write unit-tests
>> >> show up on our lists every week.
>> >>
>> >> I agree we need more unit tests. At this moment there isn’t much
>> detailed
>> >> guideline or material about how to write a unit test with unified style.
>> >> For example, before 4.1.0, mocks are injected with MockComponentLocator
>> >> and these mock objects must be implemented manually. Later we bring
>> >> mockito back and simplify the test codes. Now with spring being
>> introduced,
>> >> should we also use spring test framework?
>> >
>> >
>> > Yes, that's why we need to merge javelin into master ASAP, writing unit
>> test with Spring and Mockito and Junit/TestNG, is much easier.
>> >
>> >> And for different layers, the way to implement unit tests might be
>> different,
>> >> to test a Resource class, we need a lot of mocks/stubs of external
>> library; to
>> >> test a DAO, should we prepare the database with test data?
>> >
>> >> I think we need some consensus on ' what should be unit tested' and 'the
>> >> recommend way to write a unit test'.
>> >>
>> >> 4. Easing test writing is my primary concern at the moment and I've been
>> >> refactoring marvin to accomadate a easier dsl style test composition. I
>> hope
>> >> to have a working test case sometime next week.
>> >>
>> >> I haven’t followed this for a while, and I don’t use devCloud . Are
>> there any
>> >> documents describing how to run Marvin test with simulator/hypervisors?
>> >>
>> >> Regards
>> >> Mice
>> >>
>> >> -----Original Message-----
>> >> From: prasanna [mailto:srivatsav.prasanna@gmail.com] On Behalf Of
>> >> Prasanna Santhanam
>> >> Sent: Wednesday, January 30, 2013 2:02 AM
>> >> To: cloudstack-dev@incubator.apache.org
>> >> Subject: [DISCUSS] Concerns on testing and improvements
>> >>
>> >> On Tue, Jan 29, 2013 at 10:43:33AM -0500, Alex Huang wrote:
>> >> > > > It is also true CloudStack does not have a good automated
>> >> > > > regression test suite to make sure a check-in like this does
not
>> >> > > > break some other features in CloudStack. But lack of a thorough
>> >> > > > automated regression suite a problem with CloudStack in general.
>> >> We've let in other big changes in this release.
>> >> > > > I know developers who wrote this check-in have thoroughly
>> >> > > > regression
>> >> > > tested
>> >> > > > to the best of their abilities.
>> >> > > >
>> >> > > >
>> >> > > This is  a point I tried to make to a few folks before CloudStack
>> >> > > came here. If you want to provide a smooth path to committership
you
>> >> > > need a solid regression testing framework (with near total coverage)
>> >> > > and an environment. Feeling blind and vulnerable to big changes
like
>> >> > > this limits trust and we can't have a lack of trust. If you want
>> >> > > this project growing faster and healthier regression testing is
>> key: it has a
>> >> social impact.
>> >> > >
>> >> >
>> >> > +1  Again Agree on this.  CloudStack has been lacking a good
>> >> > automated regression tests.  We need to start doing something about
>> >> > it.
>> >>
>> >> This warrants its own thread (which unfortunately has become a common
>> >> feature on our lists)
>> >>
>> >> IMO - cloudstack is and remains largely untestable! I've often seen
>> these
>> >> requests for more tests spring (no pun intended) up when a large code
>> >> merge is called for. Happened with api_refactoring as well. But the
>> truth of
>> >> the matter is "tests are an afterthought".
>> >>
>> >> If you don't believe me:
>> >>
>> >> 1) One only needs look at the list of features waiting on ipclearance
>> without
>> >> much to call in terms of tests. Feature is developed - testing can come
>> later.
>> >>
>> >> 2) More and more requests from contributors on how to write unit-tests
>> >> show up on our lists every week.
>> >>
>> >> 3) Tests cannot magically appear unless there is a concerted effort in
>> testing
>> >> right from feature conception which is lacking.
>> >>
>> >> There's another fundamental problem. Traditionally cloudstack
>> development
>> >> has happened separate from testing processes. Code is 'thrown-over-the-
>> >> wall' to be tested which is not very different from how things are
>> today. With
>> >> that attitude I only see test playing 'catch-up' to the almost
>> *maddening*
>> >> pace of development.
>> >>
>> >>
>> >> And now for some solutions:
>> >>
>> >> 1. Start test development alongside feature development. Attach a test
>> >> engineer if possible to every feature branch. Make unit-test writing
>> >> mandatory for both committers and non-committers.
>> >>
>> >> 2. I've been working with some qa folk within my company to address the
>> >> issue more proactively. They (being non-committers) look forward to
>> start
>> >> working on github [1] along with some of our engineers who wrote the
>> >> original integration tests. The test activity will proceed in the same
>> feature
>> >> branch as being used by the developer. And will be connected to
>> appropriate
>> >> infrastructure. But likely this will happen only after 4.1
>> >>
>> >> 3. Cloudstack needs massive infrastructure, build and packaging pieces
>> to
>> >> come together for doing live provider testing. While I got this up for
>> 4.0 with
>> >> a live environment, this hasn't received much love on master. Over the
>> past
>> >> couple days I've been testing the replicability of this infrastructure
>> and have
>> >> been moving the pieces to builds.a.o
>> >>
>> >> 4. Easing test writing is my primary concern at the moment and I've been
>> >> refactoring marvin to accomadate a easier dsl style test composition. I
>> hope
>> >> to have a working test case sometime next week.
>> >>
>> >> Much of this has come too little, too late. However I beseech others to
>> pitch
>> >> in with ideas and contributions on improving testability on the whole
>> to ease
>> >> this pain for the forthcoming releases.
>> >>
>> >> [1] https://github.com/cloudstack-qa
>> >>
>> >> --
>> >> Prasanna.,
>> >> PS: Forgot to break the thread earlier in haste.
>>

Mime
View raw message