cloudstack-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Daan Hoogland <daan.hoogl...@gmail.com>
Subject Re: [DISCUSS] Breaking out Marvin from CloudStack
Date Fri, 04 Oct 2013 12:51:09 GMT
Prasanna,

I share Chips concern. But let me start by saying that given the effort
Citrix puts in the integration testing, making it easy for your guys should
be somewhat a priority. However, can we do the separation is such a way
that integration tests as part of plugin and other enhancements is still
trivial? for instance make the maven scripts checkout a stable marvin
version and run tests using this framework. What Chip wants, i think, is
that we enforce a integration test py in every contribution not only to
validate the feature but also to ensure its continued functioning while the
contributors may have wondered off in their $dayjob.

I followed the reviews submitted and know that the ones in the marvin
corner of the world are a big chunk of it ans have few reviewers/committers
in comparison. Can your refactor be organized so that integration tests can
go in the marvin and in the cloudstack repo?

thanks,
Daan


On Fri, Oct 4, 2013 at 2:27 PM, Prasanna Santhanam <tsp@apache.org> wrote:

> I'll summarize and address the concerns raised so far.
>
> Marvin has been in this repo for a long time for us to start writing
> tests. The only tests I've seen coming are from a specific set of
> people focussed on QA efforts. I want to reduce the impediment for
> people who are writing tests *today*. Those looking to get started in
> the near future won't have any new learning to do, just that their
> code goes in an alternate repo that is pointed to right
> infrastructure.
>
> Automated testing also works in a push-to-production style very often.
> Testers need to run their tests on a deployed environment(s) quickly
> to be able to ensure it is valid and passes. By making them go through
> reviewboard each time for each test we massively slow down the
> process. (tons of fixes to tests are on rb today, not just new tests).
> We don't know if they run until they run on the environment.
>
> Reason for tests and framework to go together is simple.  If I go look
> at the jclouds repository today I find tests for rackspace cloud,
> openstack cloud, cloudstack cloud, euca clouds in the jclouds
> repository and not in the respective provider/project repository. A
> newcomer to the marvin repository will be someone interested in
> writing tests and he will also thus be able to find tests in the
> marvin repository.
>
> This also allows for more heterogenous testing of cloudstack. No one
> needs to be tied down to a framework / tool to write integration
> tests. If python is not your forte, use Chip's ruby client, or perhaps
> in the near future Chiradeep's stackmate to write your test, or even
> jclouds.
>
> Now the question of supporting older version of marvin against newer
> versions of cloudstack. Marvin now fully auto-generates itself (see
> the design in the proposal) based on endpoint. So you have the
> marvin version that will work with your endpoint only. As for being
> backwards compatible (also addressed in the design doc) - no old tests
> are broken, they will still run perfectly fine.
>
> The infrastructure (currently) only looks at the changes in the test
> directory before performing a run. It doesn't care whether server/ was
> changed or plugins/x/y/z was changed. That's because the tests are
> unrelated to what is in the rest of the repository. In fact you can't
> even run them without a deployed cloud. So I don't see why idle code
> should lie in the repo.
>
> Integration tests are essential, they will keep coming as long as
> Citrix QA is invested in the effort, but they need to come faster into
> the repos and that will be addressed by the separation IMO. Managing
> the feature submitted to cloudstack against tests submitted to marvin
> is not a hard thing to do. We simply mirror the release branches in
> marvin and submit tests there. In fact I wonder why we didn't have
> this question when docs were separated? It doesn't work any
> differently really.
>
> What I would like to see provided by CloudStack is the ability to
> upgrade all our test environments, staging environments, UATs, what
> have you in continuous integration and have tests run on the
> upgraded setup. That allows incrementally testing CloudStack the way
> users do it. The current design of installing everything from scratch,
> redoing the testbed for each automated test run is mostly a workaround
> for that inability. If we had this ability tests written in marvin can
> be run against live setups at all times as and when features merge to
> master.
>
> It has always been my goal to reduce the barrier to writing tests for
> cloudstack and that's entirely the goal of the refactor as well. So I
> hope I have spent my time well. Since I can't be after people to write
> tests, and setup mandates, I aim to address the technical difficulty
> in writing tests through this refactor and separation.
>
> PS: David, by hosted docs I mean something like
> http://pythonhosted.org/cloudmonkey. Lives in the repo but read from
> the webpage.
>
> --
> Prasanna.,
>
> ------------------------
> Powered by BigRock.com
>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message