cloudstack-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Remi Bergsma <RBerg...@schubergphilis.com>
Subject Re: [PROPOSAL] Freeze everything until we get CI
Date Thu, 28 Jan 2016 08:56:42 GMT
Hi Paul,

Indeed not all Marvin tests are generic enough and can be run on all environments. If you
use this [1] collection of ~85 tests it should work well as we tested the PRs with these.
Sure we should add more, but it’s probably way faster if you’d use these and work from
there, instead of trying to get them all to work.

Regards,
Remi

[1] https://github.com/schubergphilis/MCT-shared/blob/master/helper_scripts/cloudstack/run_marvin_router_tests.sh
(the name of the file by now is a bit mis-leading)



From: Paul Angus <paul.angus@shapeblue.com<mailto:paul.angus@shapeblue.com>>
Reply-To: "dev@cloudstack.apache.org<mailto:dev@cloudstack.apache.org>" <dev@cloudstack.apache.org<mailto:dev@cloudstack.apache.org>>
Date: Thursday 28 January 2016 09:29
To: "dev@cloudstack.apache.org<mailto:dev@cloudstack.apache.org>" <dev@cloudstack.apache.org<mailto:dev@cloudstack.apache.org>>
Subject: RE: [PROPOSAL] Freeze everything until we get CI

Hi,

I mentioned last week I am working with a few of my colleagues to develop a repo of the Ansible
plays to be able deploy both Marvin and virtualised environments - (relatively) simply and
then run Marvin against that environment.

As I said, we have a load of work to do to get to the bottom of issues, as currently a large
proportion of the 250 Marvin tests fail in my 'early' builds.

Once we triage issues into Marvin build/Marvin Tests/CloudStack Env/CloudStack Code, we'll
be in better shape to see where we are.

Our intention is to not only share our approach and code, but also (with everyone's blessing)
to work to integrate it into the infra that Citrix donated to Apache and allow anyone to ask
this 'communal marvin' to generate a specified environment and run any of the integration
tests against master or any given PR....

We are actively working on this, but it is going to take a little time....




[ShapeBlue]<http://www.shapeblue.com>
Paul Angus
VP Technology   ,       ShapeBlue


d:      +44 203 617 0528 | s: +44 203 603 0540<tel:+44%20203%20617%200528%20|%20s:%20+44%20203%20603%200540>
    |      m:      +44 7711 418784<tel:+44%207711%20418784>

e:      paul.angus@shapeblue.com | t: @cloudyangus<mailto:paul.angus@shapeblue.com%20|%20t:%20@cloudyangus>
     |      w:      www.shapeblue.com<http://www.shapeblue.com>

a:      53 Chandos Place, Covent Garden London WC2N 4HS UK


[cid:imagec996fc.png@836cfe96.47bdedb6]


Shape Blue Ltd is a company incorporated in England & Wales. ShapeBlue Services India
LLP is a company incorporated in India and is operated under license from Shape Blue Ltd.
Shape Blue Brasil Consultoria Ltda is a company incorporated in Brasil and is operated under
license from Shape Blue Ltd. ShapeBlue SA Pty Ltd is a company registered by The Republic
of South Africa and is traded under license from Shape Blue Ltd. ShapeBlue is a registered
trademark.
This email and any attachments to it may be confidential and are intended solely for the use
of the individual to whom it is addressed. Any views or opinions expressed are solely those
of the author and do not necessarily represent those of Shape Blue Ltd or related companies.
If you are not the intended recipient of this email, you must neither take any action based
upon its contents, nor copy or show it to anyone. Please contact the sender if you believe
you have received this email in error.




-----Original Message-----
From: Bharat Kumar [mailto:bharat.kumar@citrix.com]
Sent: 28 January 2016 14:45
To: <dev@cloudstack.apache.org<mailto:dev@cloudstack.apache.org>> <dev@cloudstack.apache.org<mailto:dev@cloudstack.apache.org>>
Subject: Re: [PROPOSAL] Freeze everything until we get CI

Hi,

I agree with Remi on the hurdles he mentioned. It is difficult to integrate 3rd party CI,
If someone wants to implement their own CI, the link below gives one way to do it.
https://cwiki.apache.org/confluence/display/CLOUDSTACK/Test+bed+orchestrator+and+test+runner+to+enable+continuos+integration

Thanks,
Bharat.

On 28-Jan-2016, at 2:52 AM, Wido den Hollander <wido@widodh.nl<mailto:wido@widodh.nl><mailto:wido@widodh.nl>>
wrote:



On 01/27/2016 10:02 PM, Remi Bergsma wrote:
Hi all,

We should keep the simple approach that was used until now: one LGTM based on code review
and one LGTM based on integration tests (that’s not the same as 2xLGTM).

If we care about master stability, every change has to be tested for regression. Period. Things
may look OK, but still break something else in an unexpected way. IMHO, giving up on it would
be a shame.

In a perfect world, CI would automate this for us and run it against every PR. Current impediments
I see for proper CI on every PR:
- there is no community hardware to run integration tests on (there was supposed to be something)
- no support from Apache Infra: need Github write access (also to hook 3rd party CI). It’s
nowhere close.
- the community has very few people that are able to run the integration tests on their own
laptop/hardware and post the results on PRs


We tried to address points 1 and 2 and I propose to give up on them.

If all devs would stop developing for a couple of days and make sure they have a box that
can run KVM, then clone the code we use [1] it should be pretty straight forward to run the
integration tests. We choose to make it all virtual and used nested-virtualisation, but that
is optional. A simple Intel Nuc or similar will do.


Yes, that's great code. I still have to master it, but the README should get us there.

I still need to create one which also spins up a Ceph cluster. Probably a good thing to do
now ;)

Wido

People can then test the PRs that they find interesting and post results, after which it can
be merged. Distribute it. Share the load.

This is what Wilder, Miguel, Boris, Daan an I have been doing for months. Simply running the
tests. Hundreds of times. That’s why we can run a 100% Mission Critical Cloud close to master
branch at Schuberg Philis.

Regards,
Remi

[1] https://github.com/schubergphilis/MCT-shared/






On 27/01/16 21:33, "Sebastien Goasguen" <runseb@gmail.com<mailto:runseb@gmail.com><mailto:runseb@gmail.com>>
wrote:


On Jan 27, 2016, at 9:25 PM, Wido den Hollander <wido@widodh.nl<mailto:wido@widodh.nl><mailto:wido@widodh.nl>>
wrote:



On 01/27/2016 09:18 PM, Sebastien Goasguen wrote:
Folks,

How about we freeze our repo entirely until we get proper CI in place.

Seems to me all the hard work from Remi and co could be lost if we start committing again.

Now Travis is not running, Jenkins fails all the time and nobody cares…

So how about we figure out CI now ? and not do anything else.


I think forces have to be combined to make this work.

Questions which come to mind: Who runs Jenkins? Do we need a additional slave?

I haven't figured out the Integration tests completely personally.


In an ideal case, PR should trigger tests totally distributed on everyone’s own hardware.
Then tests would report back on the PR.
Only when all are green can we merge.

there is an issue with creating triggers in github on our own, but I think that’s what we
should aspire to.

for instance, how can pcextreme automate its testing and report back on each PR ?

Wido

-Sebastien

Find out more about ShapeBlue and our range of CloudStack related services:
IaaS Cloud Design & Build<http://shapeblue.com/iaas-cloud-design-and-build//> |
CSForge – rapid IaaS deployment framework<http://shapeblue.com/csforge/>
CloudStack Consulting<http://shapeblue.com/cloudstack-consultancy/> | CloudStack Software
Engineering<http://shapeblue.com/cloudstack-software-engineering/>
CloudStack Infrastructure Support<http://shapeblue.com/cloudstack-infrastructure-support/>
| CloudStack Bootcamp Training Courses<http://shapeblue.com/cloudstack-training/>
Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message