cloudstack-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Nux! <>
Subject Re: [Discuss] Flexible CI/CD - Trillian
Date Tue, 02 Feb 2016 11:13:51 GMT
Hi Paul,

This is great news, good job, naming apart. :-)

Is ESXi a hard requirement on running Trillian?

Sent from the Delta quadrant using Borg technology!


----- Original Message -----
> From: "Paul Angus" <>
> To:
> Cc:
> Sent: Tuesday, 2 February, 2016 10:49:46
> Subject: [Discuss] Flexible CI/CD - Trillian

> Project: Trillian
> We have been working on PoC of a CI environment design which will:
> ·         Provide fast build or rebuild of environments for testing.
> ·         Enable multiple independent concurrent builds
> ·         Be available on-demand through automation or individual request.
> ·         Be capable of fully utilising all available hardware
> ·         Flexible enough to be used to build super-realistic development
> environments.
> We intend to contribute and maintain our work within the Apache repos. However,
> we are currently building the POC, figuring out the requirements (and quirks)
> of the individual pieces, before pushing something concrete for to the
> community to review.
> We envision that Trillian would cater for a number of use cases:
> 1.       CloudStack community integration testing of master against multiple
> deployment scenarios (using ASF infra)
> 2.       CloudStack community integration testing of PRs against multiple
> deployment scenarios (using ASF infra)
> 3.       Organisations/individuals running the full suites of tests available in
> Marvin against any physical environment they have.
> 4.       Organisations/individuals deploying and running the full suites of
> tests available in Marvin against virtualised infrastructures which can be
> deployed by Marvin.
> As we intend Trillian to test multiple environments concurrently, we use nested
> virtualization on ESXi hosts (our testing has shown that this is the only
> hypervisor which can support the nested virtualisation of all other hypervisors
> with reasonable performance). We use Ansible to deploy and configure all
> aspects of the build as this will greatly lower the barrier to entry for
> independent testers.
> We use CloudStack to provision the management server and virtualised (nested)
> hosts on the physical hosts. We are creating Ansible playbooks and roles which
> can:
> 1.       Create guest instances using Rene’s Ansible 2.0 CloudStack modules - a
> Marvin VM, a Mgmt Server (CentOS or Ubuntu), any number of compute hosts (KVM,
> vSphere or XenServer. Hyper-V later)
> 2.       Configure hosts (inc. installing the relevant CloudStack agent where
> required)
> 3.       Install required ACS packages on management server
> 4.       Configure a zone (including adding the compute hosts) via Marvin.
> 5.       Run the required Marvin tests.
> 6.       Return the results
> We may need to propose enhancements to Marvin in order to sync the configuration
> of hosts with the configuration used by Marvin.
> Using virtualised test environments, we can have multiple test scenarios running
> concurrently. To do this we have found that it is necessary to create pools or
> ranges of VLANs and IP addresses and allocate them to environments.  So for any
> given physical environment which will be used for testing in, we take the total
> range(s) of IPs and VLANs available and carve them into non-overlapping chunks
> suitable for concurrent use as mgmt, public and guest networks.  These are
> stored in a MariaDB database.  When a range is being used in a testing
> environment, that range is marked as ‘inuse’ in the database.  When creating a
> test environment, Trillian looks in the database for the next available VLAN
> range, the next available public IP range and so on.  The returned values are
> used to populate a Marvin cfg file which in turn will be used to both build the
> environment and when running the Marvin testing.  When the virtualised infra is
> cleaned up, the database will be updated to reflect that the used ranges are
> available again.
> This initiative has only recently been started, and as stated earlier we are
> currently figuring out the requirements (and quirks) of the individual pieces
> and looking for the most suitable wrapper to glue it all together.
> Also I have found that Marvin requires a little work to make the output more
> meaningful/readable (especially in the case of errors and exceptions) and to
> make it a little more intelligent about the tests it can/can’t run based on the
> chosen infrastructure components. I have also found unreachable or very slow
> ISO and template paths hardcoded into Marvin or individual tests.
> We plan to enhance tests to address these issues and also reduce runtimes where
> possible.
> We will continue to our work on the PoC and share the results.
> We’d like to get some community agreement around the goals and use cases which
> we are planning to satisfy before we get too far down the road of our PoC. So
> please share your thoughts.
> [ShapeBlue]<>
> Paul Angus
> VP Technology   ,       ShapeBlue
> d:      +44 203 617 0528 | s: +44 203 603
> 0540<tel:+44%20203%20617%200528%20|%20s:%20+44%20203%20603%200540>     |
> m:      +44 7711 418784<tel:+44%207711%20418784>
> e: | t:
> @cloudyangus<|%20t:%20@cloudyangus>      |
> w:<>
> a:      53 Chandos Place, Covent Garden London WC2N 4HS UK
> [cid:imagea932fa.png@ce3f6520.47a350c4]
> Shape Blue Ltd is a company incorporated in England & Wales. ShapeBlue Services
> India LLP is a company incorporated in India and is operated under license from
> Shape Blue Ltd. Shape Blue Brasil Consultoria Ltda is a company incorporated in
> Brasil and is operated under license from Shape Blue Ltd. ShapeBlue SA Pty Ltd
> is a company registered by The Republic of South Africa and is traded under
> license from Shape Blue Ltd. ShapeBlue is a registered trademark.
> This email and any attachments to it may be confidential and are intended solely
> for the use of the individual to whom it is addressed. Any views or opinions
> expressed are solely those of the author and do not necessarily represent those
> of Shape Blue Ltd or related companies. If you are not the intended recipient
> of this email, you must neither take any action based upon its contents, nor
> copy or show it to anyone. Please contact the sender if you believe you have
> received this email in error.
> Find out more about ShapeBlue and our range of CloudStack related services:
> IaaS Cloud Design & Build<>
> CSForge – rapid IaaS deployment framework<>
> CloudStack Consulting<> | CloudStack
> Software Engineering<>
> CloudStack Infrastructure
> Support<> | CloudStack
> Bootcamp Training Courses<>

View raw message