cloudstack-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Edison Su <Edison...@citrix.com>
Subject RE: [DISCUSSION] CI
Date Thu, 21 Aug 2014 17:32:58 GMT


> -----Original Message-----
> From: Pierre-Luc Dion [mailto:pdion@cloudops.com]
> Sent: Thursday, August 21, 2014 5:06 AM
> To: dev@cloudstack.apache.org
> Cc: Will Stevens
> Subject: Re: [DISCUSSION] CI
> 
> Thanks for you responses,
> 
> The revamp of marvin look promising, I'll have a closer look at it, even if I'm
> not Python fan ;).  In our side, we have close to fully automated installation
> of CloudStack, XenServer hypervisor and an initial zone creation.  Our missing


Will it be open sourced?:) 

> part is automated test cases to use it. So we could provide infrastructure and
> run tests against it for one or more  test scenarios during QA phase of release.
> Hopefully this could be integrated to the CI.
Currently http://jenkins.buildacloud.org/, is CloudStack CI portal.  If you can provide
Infrastructure to integrate with it, that will be awesome!

> 
> Would it make sense to have a sheet, maybe a wiki page that list  all
> infrastructure test scenarios and their result per version of ACS?
> for:
> xenserver, hyperv, vmware,kvm
> advanced, simplenetworking, vlan/vxlan,gre isolated, networks s3,swift,nfs,
> storage plugin
> 
> Unless we have this kind of results somewhere, I don't feel we really keep
> track of what has been tested on each release.
It's a good idea. Let me ask the team who wrote these test cases, maybe they have answers.

> 
> 
> 
> 
> *Pierre-Luc DION*
> Architecte de Solution Cloud | Cloud Solutions Architect t 855.652.5683
> 
> *CloudOps* Votre partenaire infonuagique* | *Cloud Solutions Experts
> 420 rue Guy *|* Montreal *|* Quebec *|* H3J 1S6 w cloudops.com *|* tw
> @CloudOps_
> 
> 
> 
> On Thu, Aug 21, 2014 at 2:03 AM, Prasanna Santhanam <tsp@qubole.com>
> wrote:
> 
> > I use rspec in Ruby these days and find it get complex to maintain
> > pretty quickly. It's good for dealing with REST apis and middleware
> > but with Marvin (and python in general) I find infrastructure handling
> > easier. Stack traces of BDD code is also a little bit hard to
> > troubleshoot and comprehend. I may be biased because my Python
> > expertise is better, so you can take this with a pinch of salt.
> >
> > Edison is right however, the setup in marvin is atrociously complex
> > and needs to be a lot simpler and intuitive. In most cases I don't
> > want to specify tons of args (service offering, disk offering, network
> > type) to create a simple vM that is part of every test. To this end I
> > refactored Marvin to use easily defined factories (in lines with
> > Ruby's factory_girl) and also demo-ed some code in the branch. Haven't
> > had time to maintain and revamp this honestly. So if anyone wants to
> > pick up from there, I'm happy to help.
> >
> >
> https://cwiki.apache.org/confluence/display/CLOUDSTACK/Marvin+Refactor
> >
> >
> >
> > On Thu, Aug 21, 2014 at 4:30 AM, Edison Su <Edison.su@citrix.com> wrote:
> > >
> > >
> > >> -----Original Message-----
> > >> From: Pierre-Luc Dion [mailto:pdion@cloudops.com]
> > >> Sent: Wednesday, August 20, 2014 11:20 AM
> > >> To: dev@cloudstack.apache.org; Will Stevens
> > >> Subject: [DISCUSSION] CI
> > >>
> > >> I'd like to move this discussion out of the GIT workflow thread.
> > >>
> > >> Do we have any CI strategy in place or a starting point at least?
> > > TravisCI will be a starting point for now. It tests against
> > > simulator
> > only.
> > > I am working on a solution to test against KVM(that's most of users
> > > are
> > concerned about) in a machine I created in
> > > Public cloud. It has big memory, super fast disk IO, should be a
> > > good
> > solution for KVM test. It may take few days to get it work.
> > > They are both for simple test, and in small scale.
> > >>
> > >> I've never worked with marvin I from the small understanding that I
> > >> have from it, it might not be the right too. I don't know.
> > >>
> > >> I've spend some times recently with test-kitchen and serverspec and
> > found
> > >> that quite promising as test platform for "real cloudstack
> > >> deployment
> > test
> > >> scenarios". By using test framework like Rspec the test output is
> > >> clean
> > to
> > >> read for human and test scenarios seams quite simple to write.
> > >>
> > >> I've started a rspec lib for cloudstack to enhance serverspec for
> > CloudStack
> > >> tests (cloudstack-rspec).  I'm wondering if it would be usable in
> > future CI
> > >> system for CloudStack...
> > > I am familiar with Specs,  like Rspec, but in scala. I agree with
> > > you
> > that the BDD style test case
> > > Is easier to read/write. It's good to have, but I think before we
> > > move
> > to new test framework, we should figure out the issues
> > > We have today in Marvin, so that we can take effort to address them,
> > either fixing Marvin, or switch to a new test framework.
> > > The issue I have in current Marvin, is that it still needs to write
> > > a
> > lot of setup code for each test case, which is complicated enough.
> > Let's take an Marvin test case as an example:
> > https://git-wip-us.apache.org/repos/asf?p=cloudstack.git;a=blob;f=test
> >
> /integration/smoke/test_vm_life_cycle.py;h=57e47c24838ffc2ec5e9af4ad38
> > 72d5e27342d91;hb=HEAD
> > >
> > > Like this kind of code in test case setup code:
> > >
> > > # Get Zone, Domain and templates
> > >         cls.domain = get_domain(cls.apiclient)
> > >         cls.zone = get_zone(cls.apiclient, testClient.getZoneForTests())
> > >         cls.services['mode'] = cls.zone.networktype
> > >
> > >         #If local storage is enabled, alter the offerings to use
> > localstorage
> > >         #this step is needed for devcloud
> > >         if cls.zone.localstorageenabled == True:
> > >             cls.services["service_offerings"]["tiny"]["storagetype"]
> > > =
> > 'local'
> > >
> > > cls.services["service_offerings"]["small"]["storagetype"] =
> > 'local'
> > >
> > > cls.services["service_offerings"]["medium"]["storagetype"] =
> > 'local'
> > >
> > > this kind of information should be provided by test framework
> > automatically, instead of explicitly set in every test case.
> > > Do you have other concerns about current Marvin, other than test style?
> > >
> > >
> > >>
> > >> Also, having a CI in place for cloudstack is quite of a huge deal
> > >> if we
> > want to
> > >> validate all kind of deployment scenario, networks, hypervisor
> > >> version
> > and
> > >> guest OS version.
> > >
> > >
> > > Can't agree with you more.
> > >
> > >>
> > >>
> > >> For those that have some knowledge on the CI subject in CloudStack
> > please
> > >> update the community. I don't feel alone being lost  about this
> > CloudStack
> > >> topic.
> > >
> > > There is an update-to-date write up regarding to Marvin:
> > https://cwiki.apache.org/confluence/display/CLOUDSTACK/Marvin+-
> +Testin
> > g+with+Python
> > > Marvin does two things, one is to call cloudstack mgt API to add
> > resources into CloudStack mgt server, such as hypervisors, primary
> > storages, networks, and secondary storages etc.
> > > Another one is to run test cases under tools/integration/smoke or
> > > any
> > other python test cases.
> > > It doesn't cover how to deploy hypervisors on real HW, how to setup
> > primary storages, secondary storage etc. That's something like
> > test-kitchen and serverspec can help, I think.
> > >
> > >>
> > >> Thanks,
> > >>
> > >>
> > >> Pierre-Luc
> >
> >
> >
> > --
> > Prasanna.,
> >
Mime
View raw message