cloudstack-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Prasanna Santhanam <...@apache.org>
Subject Re: [DISCUSS] Breaking out Marvin from CloudStack
Date Fri, 04 Oct 2013 12:27:58 GMT
I'll summarize and address the concerns raised so far.

Marvin has been in this repo for a long time for us to start writing
tests. The only tests I've seen coming are from a specific set of
people focussed on QA efforts. I want to reduce the impediment for
people who are writing tests *today*. Those looking to get started in
the near future won't have any new learning to do, just that their
code goes in an alternate repo that is pointed to right
infrastructure. 

Automated testing also works in a push-to-production style very often.
Testers need to run their tests on a deployed environment(s) quickly
to be able to ensure it is valid and passes. By making them go through
reviewboard each time for each test we massively slow down the
process. (tons of fixes to tests are on rb today, not just new tests).
We don't know if they run until they run on the environment.

Reason for tests and framework to go together is simple.  If I go look
at the jclouds repository today I find tests for rackspace cloud,
openstack cloud, cloudstack cloud, euca clouds in the jclouds
repository and not in the respective provider/project repository. A
newcomer to the marvin repository will be someone interested in
writing tests and he will also thus be able to find tests in the
marvin repository.

This also allows for more heterogenous testing of cloudstack. No one
needs to be tied down to a framework / tool to write integration
tests. If python is not your forte, use Chip's ruby client, or perhaps
in the near future Chiradeep's stackmate to write your test, or even
jclouds. 

Now the question of supporting older version of marvin against newer
versions of cloudstack. Marvin now fully auto-generates itself (see
the design in the proposal) based on endpoint. So you have the
marvin version that will work with your endpoint only. As for being
backwards compatible (also addressed in the design doc) - no old tests
are broken, they will still run perfectly fine. 

The infrastructure (currently) only looks at the changes in the test
directory before performing a run. It doesn't care whether server/ was
changed or plugins/x/y/z was changed. That's because the tests are
unrelated to what is in the rest of the repository. In fact you can't
even run them without a deployed cloud. So I don't see why idle code
should lie in the repo.

Integration tests are essential, they will keep coming as long as
Citrix QA is invested in the effort, but they need to come faster into
the repos and that will be addressed by the separation IMO. Managing
the feature submitted to cloudstack against tests submitted to marvin
is not a hard thing to do. We simply mirror the release branches in
marvin and submit tests there. In fact I wonder why we didn't have
this question when docs were separated? It doesn't work any
differently really.

What I would like to see provided by CloudStack is the ability to
upgrade all our test environments, staging environments, UATs, what
have you in continuous integration and have tests run on the
upgraded setup. That allows incrementally testing CloudStack the way
users do it. The current design of installing everything from scratch,
redoing the testbed for each automated test run is mostly a workaround
for that inability. If we had this ability tests written in marvin can
be run against live setups at all times as and when features merge to
master.

It has always been my goal to reduce the barrier to writing tests for
cloudstack and that's entirely the goal of the refactor as well. So I
hope I have spent my time well. Since I can't be after people to write
tests, and setup mandates, I aim to address the technical difficulty
in writing tests through this refactor and separation.  

PS: David, by hosted docs I mean something like
http://pythonhosted.org/cloudmonkey. Lives in the repo but read from
the webpage.

-- 
Prasanna.,

------------------------
Powered by BigRock.com


Mime
View raw message