cloudstack-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Pierre-Luc Dion <pd...@cloudops.com>
Subject Re: [DISCUSSION] CI
Date Thu, 21 Aug 2014 12:05:45 GMT
Thanks for you responses,

The revamp of marvin look promising, I'll have a closer look at it, even if
I'm not Python fan ;).  In our side, we have close to fully automated
installation of CloudStack, XenServer hypervisor and an initial zone
creation.  Our missing part is automated test cases to use it. So we could
provide infrastructure and run tests against it for one or more  test
scenarios during QA phase of release. Hopefully this could be integrated to
the CI.

Would it make sense to have a sheet, maybe a wiki page that list  all
infrastructure test scenarios and their result per version of ACS?
for:
xenserver, hyperv, vmware,kvm
advanced, simplenetworking, vlan/vxlan,gre isolated, networks
s3,swift,nfs, storage plugin

Unless we have this kind of results somewhere, I don't feel we really keep
track of what has been tested on each release.




*Pierre-Luc DION*
Architecte de Solution Cloud | Cloud Solutions Architect
t 855.652.5683

*CloudOps* Votre partenaire infonuagique* | *Cloud Solutions Experts
420 rue Guy *|* Montreal *|* Quebec *|* H3J 1S6
w cloudops.com *|* tw @CloudOps_



On Thu, Aug 21, 2014 at 2:03 AM, Prasanna Santhanam <tsp@qubole.com> wrote:

> I use rspec in Ruby these days and find it get complex to maintain
> pretty quickly. It's good for dealing with REST apis and middleware
> but with Marvin (and python in general) I find infrastructure handling
> easier. Stack traces of BDD code is also a little bit hard to
> troubleshoot and comprehend. I may be biased because my Python
> expertise is better, so you can take this with a pinch of salt.
>
> Edison is right however, the setup in marvin is atrociously complex
> and needs to be a lot simpler and intuitive. In most cases I don't
> want to specify tons of args (service offering, disk offering, network
> type) to create a simple vM that is part of every test. To this end I
> refactored Marvin to use easily defined factories (in lines with
> Ruby's factory_girl) and also demo-ed some code in the branch. Haven't
> had time to maintain and revamp this honestly. So if anyone wants to
> pick up from there, I'm happy to help.
>
> https://cwiki.apache.org/confluence/display/CLOUDSTACK/Marvin+Refactor
>
>
>
> On Thu, Aug 21, 2014 at 4:30 AM, Edison Su <Edison.su@citrix.com> wrote:
> >
> >
> >> -----Original Message-----
> >> From: Pierre-Luc Dion [mailto:pdion@cloudops.com]
> >> Sent: Wednesday, August 20, 2014 11:20 AM
> >> To: dev@cloudstack.apache.org; Will Stevens
> >> Subject: [DISCUSSION] CI
> >>
> >> I'd like to move this discussion out of the GIT workflow thread.
> >>
> >> Do we have any CI strategy in place or a starting point at least?
> > TravisCI will be a starting point for now. It tests against simulator
> only.
> > I am working on a solution to test against KVM(that's most of users are
> concerned about) in a machine I created in
> > Public cloud. It has big memory, super fast disk IO, should be a good
> solution for KVM test. It may take few days to get it work.
> > They are both for simple test, and in small scale.
> >>
> >> I've never worked with marvin I from the small understanding that I have
> >> from it, it might not be the right too. I don't know.
> >>
> >> I've spend some times recently with test-kitchen and serverspec and
> found
> >> that quite promising as test platform for "real cloudstack deployment
> test
> >> scenarios". By using test framework like Rspec the test output is clean
> to
> >> read for human and test scenarios seams quite simple to write.
> >>
> >> I've started a rspec lib for cloudstack to enhance serverspec for
> CloudStack
> >> tests (cloudstack-rspec).  I'm wondering if it would be usable in
> future CI
> >> system for CloudStack...
> > I am familiar with Specs,  like Rspec, but in scala. I agree with you
> that the BDD style test case
> > Is easier to read/write. It's good to have, but I think before we move
> to new test framework, we should figure out the issues
> > We have today in Marvin, so that we can take effort to address them,
> either fixing Marvin, or switch to a new test framework.
> > The issue I have in current Marvin, is that it still needs to write a
> lot of setup code for each test case, which is complicated enough. Let's
> take an Marvin test case as an example:
> https://git-wip-us.apache.org/repos/asf?p=cloudstack.git;a=blob;f=test/integration/smoke/test_vm_life_cycle.py;h=57e47c24838ffc2ec5e9af4ad3872d5e27342d91;hb=HEAD
> >
> > Like this kind of code in test case setup code:
> >
> > # Get Zone, Domain and templates
> >         cls.domain = get_domain(cls.apiclient)
> >         cls.zone = get_zone(cls.apiclient, testClient.getZoneForTests())
> >         cls.services['mode'] = cls.zone.networktype
> >
> >         #If local storage is enabled, alter the offerings to use
> localstorage
> >         #this step is needed for devcloud
> >         if cls.zone.localstorageenabled == True:
> >             cls.services["service_offerings"]["tiny"]["storagetype"] =
> 'local'
> >             cls.services["service_offerings"]["small"]["storagetype"] =
> 'local'
> >             cls.services["service_offerings"]["medium"]["storagetype"] =
> 'local'
> >
> > this kind of information should be provided by test framework
> automatically, instead of explicitly set in every test case.
> > Do you have other concerns about current Marvin, other than test style?
> >
> >
> >>
> >> Also, having a CI in place for cloudstack is quite of a huge deal if we
> want to
> >> validate all kind of deployment scenario, networks, hypervisor version
> and
> >> guest OS version.
> >
> >
> > Can't agree with you more.
> >
> >>
> >>
> >> For those that have some knowledge on the CI subject in CloudStack
> please
> >> update the community. I don't feel alone being lost  about this
> CloudStack
> >> topic.
> >
> > There is an update-to-date write up regarding to Marvin:
> https://cwiki.apache.org/confluence/display/CLOUDSTACK/Marvin+-+Testing+with+Python
> > Marvin does two things, one is to call cloudstack mgt API to add
> resources into CloudStack mgt server, such as hypervisors, primary
> storages, networks, and secondary storages etc.
> > Another one is to run test cases under tools/integration/smoke or any
> other python test cases.
> > It doesn't cover how to deploy hypervisors on real HW, how to setup
> primary storages, secondary storage etc. That's something like test-kitchen
> and serverspec can help, I think.
> >
> >>
> >> Thanks,
> >>
> >>
> >> Pierre-Luc
>
>
>
> --
> Prasanna.,
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message