cloudstack-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Will Stevens <wstev...@cloudops.com>
Subject Re: Important Pending Items
Date Wed, 02 Mar 2016 14:06:00 GMT
Thanks Sanjeev, I did not realize that the whole simulator was tested.  Ok
that is good...  :)

*Will STEVENS*
Lead Developer

*CloudOps* *| *Cloud Solutions Experts
420 rue Guy *|* Montreal *|* Quebec *|* H3J 1S6
w cloudops.com *|* tw @CloudOps_

On Wed, Mar 2, 2016 at 12:15 AM, Sanjeev N <sanjeev@apache.org> wrote:

> Hi Steve,
>
> Currently travis runs all simulator tests(marvin tests which does not
> require any hardware, tagged with requierd_hardware=false) on every PR. If
> any test fails it updates the PR and marks it red. IMO apart from 2 LGTMs
> travis test pass is also mandatory for PR merge.
>
> Thanks,
> Sanjeev N
> Chief Product Engineer@Accelerite
>
> On Wed, Mar 2, 2016 at 3:13 AM, Will Stevens <wstevens@cloudops.com>
> wrote:
>
> > Awesome, thanks Remi.  I will review this.
> >
> > *Will STEVENS*
> > Lead Developer
> >
> > *CloudOps* *| *Cloud Solutions Experts
> > 420 rue Guy *|* Montreal *|* Quebec *|* H3J 1S6
> > w cloudops.com *|* tw @CloudOps_
> >
> > On Tue, Mar 1, 2016 at 4:29 PM, Remi Bergsma <
> RBergsma@schubergphilis.com>
> > wrote:
> >
> > > Hi Will,
> > >
> > > We used this to test hundreds of PRs:
> > > https://github.com/schubergphilis/MCT-shared/. I can talk you through
> it
> > > sometime next week or so if you want.
> > >
> > > Regards,
> > > Remi
> > >
> > >
> > > On 01/03/16 21:50, "williamstevens@gmail.com on behalf of Will
> Stevens"
> > <
> > > williamstevens@gmail.com on behalf of wstevens@cloudops.com> wrote:
> > >
> > > >Thanks Paul.  Great to hear you are planning to make everything
> > available
> > > >and you are willing to take contributions from the community.
> > > >
> > > >I am currently trying to get up to speed on everything that has been
> > done
> > > >and all the different pieces in play.  I would like to have a CI in
> > place
> > > >(even if it is only locally on my hardware) for testing PRs for 4.9.
> My
> > > >focus right now is solving the CI problem, so I am interested in any
> > > >developments on this front.
> > > >
> > > >We have Travis doing smoke tests on every PR right now.  Travis is
> free
> > > for
> > > >open source projects.  Is there a reason we are not doing more
> extensive
> > > >tests with Travis since it is already integrated into github?  Like
> > maybe
> > > >running the simulator on every PR?  I think this could be a good first
> > > step
> > > >to weed out some of the PRs that are likely to fail a full CI run.
> > > >
> > > >Cheers,
> > > >
> > > >*Will STEVENS*
> > > >Lead Developer
> > > >
> > > >*CloudOps* *| *Cloud Solutions Experts
> > > >420 rue Guy *|* Montreal *|* Quebec *|* H3J 1S6
> > > >w cloudops.com *|* tw @CloudOps_
> > > >
> > > >On Tue, Mar 1, 2016 at 3:40 PM, Paul Angus <paul.angus@shapeblue.com>
> > > wrote:
> > > >
> > > >> Hi Will,
> > > >>
> > > >> It will be 100% open source. The $dayjob has blocked the team from
> > > working
> > > >> on it recently. I'd like to get some of our ideas into the
> CloudStack
> > > wiki
> > > >> this week before I go on leave for a week.
> > > >> Rohit has created a repo for us:
> > https://github.com/shapeblue/Trillian
> > > >> (which looks a bit sad at the moment - I hope to push our Ansible
> > roles
> > > >> for building mgmt. servers and Marvin boxes into it this week also)
> > > >>
> > > >> Internally we're planning a two week sprint on it starting 14th
> March.
> > > >> Having to dip in and out of this effort is making it very difficult
> to
> > > get
> > > >> anywhere
> > > >>
> > > >> I think the biggest part of the work will be to get Marvin up to
> > > scratch,
> > > >> the log output from the tests is pretty bad.
> > > >>
> > > >> We very much want this to be a community 'thing', and we're happy
to
> > > work
> > > >> with everyone/anyone to get there.
> > > >>
> > > >> Any ideas on the most effective way to have a number of people on
> > this -
> > > >> I'm all ears.
> > > >> In the meantime I'll try to make good on my promises to get our
> ideas
> > > >> visible to everyone...
> > > >>
> > > >>
> > > >>
> > > >>
> > > >> Paul Angus
> > > >> VP Technology   ,       ShapeBlue
> > > >>
> > > >>
> > > >> t:      @cloudyangus<tel:@cloudyangus>
> > > >>
> > > >> e:      paul.angus@shapeblue.com<mailto:paul.angus@shapeblue.com>
> > > >> |      w:      www.shapeblue.com<http://www.shapeblue.com>
> > > >>
> > > >>
> > > >>
> > > >>
> > > >>
> > > >> -----Original Message-----
> > > >> From: williamstevens@gmail.com [mailto:williamstevens@gmail.com] On
> > > >> Behalf Of Will Stevens
> > > >> Sent: Monday, February 29, 2016 9:52 PM
> > > >> To: dev@cloudstack.apache.org
> > > >> Subject: Re: Important Pending Items
> > > >>
> > > >> @Paul and @Bharat, thank you for the quality write-ups on this
> topic.
> > > >>
> > > >> re:Paul's Details:
> > > >> I share a lot of your thoughts on this. The hypervisor in hypervisor
> > > setup
> > > >> with the ability to test multiple environments is also core to my
> > > thinking.
> > > >> I can't find any details on Trillian, other than what is in this
> > > thread. Is
> > > >> it open source and freely available? This is being developed at
> > > ShapeBlue?
> > > >>
> > > >> re:Bharat's Details:
> > > >> This seems like it is the most likely candidate for being able to
> > offer
> > > >> value in the short term. Thank you for the work documenting it and
> > > making
> > > >> the source available. I still need to review the code. This approach
> > > will
> > > >> take quite a bit of work get setup initially, but once setup, it
> looks
> > > like
> > > >> it will do the trick. By using only freely available software makes
> it
> > > more
> > > >> accessible than some options which have been suggested in the past.
> > > >>
> > > >> My thoughts on the topic:
> > > >> - I think it is important that the CI can test all the different
> > > >> hypervisors and any hardware integrations available in the
> > environment.
> > > >> - Ideally the CI tool would be easily distributed and installed at
> > > >> multiple sites. Every company who depends on ACS will have their own
> > > >> configuration, hardware and mission critical features. If the CI can
> > be
> > > >> distributed (as independent installs), it will reduce the load on
> any
> > > >> specific site and it will enable companies who have specific
> hardware
> > in
> > > >> their direct path to success the ability to validate their use case.
> > > >> - I think a standardized CI output is also important. Ideally the
> > > summary
> > > >> of the CI run would be posted back to the PR thread. I think it is
> > also
> > > >> important to push the detailed logs to a central location so others
> > can
> > > >> review them. Ideally, links to the detailed logs would also be
> pushed
> > to
> > > >> the PR thread.
> > > >> - I think there should be multiple tiers of testing. Basic simulator
> > > >> testing and full runs. We may be able to setup simulator testing
> > using a
> > > >> Docker container. I think @pdion891 has put some work into this
> that I
> > > may
> > > >> be able to build on.
> > > >> - I think it is probably too ambitious to do CI on every PR before
> > > merge,
> > > >> but ideally we would We may have to only CI the master branch for
> now
> > > and
> > > >> revert any PR that causes it to fail.
> > > >>
> > > >> I have been digging into this quite a bit today. I will continue
> > working
> > > >> on this as it is very important to me for the 4.9 release. I would
> > like
> > > to
> > > >> setup a CI environment locally as well to help validate and
> contribute
> > > to
> > > >> the effort. I have some hardware I can throw at this, so I will be
> > > working
> > > >> to get something setup to start testing ASAP.
> > > >>
> > > >> If I can work off either (or both) of your work, I would appreciate
> > it.
> > > >>
> > > >> Cheers,
> > > >>
> > > >> *Will STEVENS*
> > > >> Lead Developer
> > > >>
> > > >> *CloudOps* *| *Cloud Solutions Experts
> > > >> 420 rue Guy *|* Montreal *|* Quebec *|* H3J 1S6 w cloudops.com *|*
> tw
> > > >> @CloudOps_
> > > >>
> > > >> On Fri, Feb 12, 2016 at 12:49 AM, Bharat Kumar <
> > bharat.kumar@citrix.com
> > > >
> > > >> wrote:
> > > >>
> > > >> > Hi Paul,
> > > >> >
> > > >> > The goals and functionality of the CI on which we are working
are
> > the
> > > >> > same (please review the FS<
> > > >> >
> > > https://cwiki.apache.org/confluence/display/CLOUDSTACK/Test+bed+orches
> > > >> > trator+and+test+runner+to+enable+continuos+integration>
> > > >> > ), The difference being the method of implementation. We are
not
> > using
> > > >> > nested virtualisation to create test environments. The work on
CI
> > > >> > began long back but unfortunately due to other priorities we
could
> > not
> > > >> > work on a continuous basis and bring it to completion.
> > > >> >
> > > >> > I agree that we should come to a consensus on what the community
> > wants
> > > >> > from the CI. IMHO the basic and immediate requirement would be
to
> > have
> > > >> > a stable system to test the PRs and post results. Once we have
> this,
> > > >> > we can talk about providing test setups to the community and
next
> > > >> > steps. I think the availability of hardware has been a big hurdle
> > > >> > (even if we use virtualisation).
> > > >> >
> > > >> > Instead of working independently on the same thing let us work
on
> > this
> > > >> > together. I will share what we have currently, we can use it,
or
> > parts
> > > >> > of it (some of it is there at
> > > >> > https://github.com/bvbharatk/cloudstack-automation, but it is
not
> > > >> > organised as in cannot be run out of the box). While
> virtualisation
> > is
> > > >> > good from the ease of orchestration and resource utilisation
point
> > of
> > > >> > view, it has other problems like compatibility/dependencies
> issues.
> > > >> > Also it might tie the CI to a particular Hypervisor like ESXi
as
> you
> > > >> have pointed out.
> > > >> >
> > > >> > So let us discuss on how to proceed further, and get things
> rolling.
> > > >> >
> > > >> > Thanks,
> > > >> > Bharat.
> > > >> >
> > > >> >
> > > >> > On 11-Feb-2016, at 9:44 PM, Paul Angus <paul.angus@shapeblue.com
> > > <mailto:
> > > >> > paul.angus@shapeblue.com>> wrote:
> > > >> >
> > > >> > Hi Bharat,
> > > >> >
> > > >> > Early last week I opened a discussion thread around the aims
of
> > > >> > Trillian which we had begun working on as a CI/CD platform for
the
> > > >> > community hardware (as well as individual use). Please could
you
> > > >> > elaborate on where there is/isn't cross-over? In terms of
> > > >> > functionality and goals rather than method. It would be great
to
> get
> > > >> > community consensus around what we want from our centralised
> CI/CD.
> > > >> >
> > > >> >
> > > >> > Project: Trillian
> > > >> > We have been working on PoC of a CI environment design which
will:
> > > >> > * Provide fast build or rebuild of environments for testing.
> > > >> > * Enable multiple independent concurrent builds
> > > >> > * Be available on-demand through automation or individual request.
> > > >> > * Be capable of fully utilising all available hardware
> > > >> > * Flexible enough to be used to build super-realistic development
> > > >> > environments.
> > > >> > We intend to contribute and maintain our work within the Apache
> > repos.
> > > >> > However, we are currently building the POC, figuring out the
> > > >> > requirements (and quirks) of the individual pieces, before pushing
> > > >> > something concrete for to the community to review.
> > > >> >
> > > >> > We envision that Trillian would cater for a number of use cases:
> > > >> > 1. CloudStack community integration testing of master against
> > multiple
> > > >> > deployment scenarios (using ASF infra) 2. CloudStack community
> > > >> > integration testing of PRs against multiple deployment scenarios
> > > >> > (using ASF infra) 3. Organisations/individuals running the full
> > suites
> > > >> > of tests available in Marvin against any physical environment
they
> > > >> > have.
> > > >> > 4. Organisations/individuals deploying and running the full suites
> > of
> > > >> > tests available in Marvin against virtualised infrastructures
> which
> > > >> > can be deployed by Marvin.
> > > >> > As we intend Trillian to test multiple environments concurrently,
> we
> > > >> > use nested virtualization on ESXi hosts (our testing has shown
> that
> > > >> > this is the only hypervisor which can support the nested
> > > >> > virtualisation of all other hypervisors with reasonable
> > performance).
> > > >> > We use Ansible to deploy and configure all aspects of the build
as
> > > >> > this will greatly lower the barrier to entry for independent
> > testers.
> > > >> > We use CloudStack to provision the management server and
> virtualised
> > > >> > (nested) hosts on the physical hosts. We are creating Ansible
> > > >> > playbooks and roles which can:
> > > >> > 1. Create guest instances using Rene's Ansible 2.0 CloudStack
> > modules
> > > >> > - a Marvin VM, a Mgmt Server (CentOS or Ubuntu), any number of
> > compute
> > > >> > hosts (KVM, vSphere or XenServer. Hyper-V later) 2. Configure
> hosts
> > > >> > (inc. installing the relevant CloudStack agent where
> > > >> > required)
> > > >> > 3. Install required ACS packages on management server 4.
> Configure a
> > > >> > zone (including adding the compute hosts) via Marvin.
> > > >> > 5. Run the required Marvin tests.
> > > >> > 6. Return the results
> > > >> > We may need to propose enhancements to Marvin in order to sync
the
> > > >> > configuration of hosts with the configuration used by Marvin.
> > > >> >
> > > >> > Using virtualised test environments, we can have multiple test
> > > >> > scenarios running concurrently. To do this we have found that
it
> is
> > > >> > necessary to create pools or ranges of VLANs and IP addresses
and
> > > >> > allocate them to environments. So for any given physical
> environment
> > > >> > which will be used for testing in, we take the total range(s)
of
> IPs
> > > >> > and VLANs available and carve them into non-overlapping chunks
> > > >> > suitable for concurrent use as mgmt, public and guest networks.
> > These
> > > >> > are stored in a MariaDB database. When a range is being used
in a
> > > >> > testing environment, that range is marked as 'inuse' in the
> > database.
> > > >> > When creating a test environment, Trillian looks in the database
> for
> > > >> > the next available VLAN range, the next available public IP range
> > and
> > > >> > so on. The returned values are used to populate a Marvin cfg
file
> > > >> > which in turn will be used to both build the environment and
when
> > > >> > running the Marvin testing. When the virtualised infra is cleaned
> > up,
> > > >> > the database will be updated to reflect that the used ranges
are
> > > >> available again.
> > > >> > This initiative has only recently been started, and as stated
> > earlier
> > > >> > we are currently figuring out the requirements (and quirks) of
the
> > > >> > individual pieces and looking for the most suitable wrapper to
> glue
> > it
> > > >> all together.
> > > >> > Also I have found that Marvin requires a little work to make
the
> > > >> > output more meaningful/readable (especially in the case of errors
> > and
> > > >> > exceptions) and to make it a little more intelligent about the
> tests
> > > >> > it can/can't run based on the chosen infrastructure components.
I
> > have
> > > >> > also found unreachable or very slow ISO and template paths
> hardcoded
> > > >> > into Marvin or individual tests.
> > > >> > We plan to enhance tests to address these issues and also reduce
> > > >> > runtimes where possible.
> > > >> >
> > > >> >
> > > >> > <http://www.shapeblue.com/>
> > > >> > Paul Angus
> > > >> > VP Technology , ShapeBlue
> > > >> >
> > > >> >
> > > >> > d: +44 203 617 0528 | s: +44 203 603 0540
> > > >> <tel:+44%20203%20617%200528%20|%20s:%20+44%20203%20603%200540>
> > > >> > | m: +44 7711 418784<tel:+44%207711%20418784>
> > > >> >
> > > >> > e: paul.angus@shapeblue.com | t: @cloudyangus<mailto:
> > > >> > paul.angus@shapeblue.com%20|%20t:%20@cloudyangus> | w:
> > > >> > www.shapeblue.com<http://www.shapeblue.com/>
> > > >> >
> > > >> > a: 53 Chandos Place, Covent Garden London WC2N 4HS UK
> > > >> >
> > > >> >
> > > >> >
> > > >> >
> > > >> >
> > > >> > Shape Blue Ltd is a company incorporated in England & Wales.
> > ShapeBlue
> > > >> > Services India LLP is a company incorporated in India and is
> > operated
> > > >> > under license from Shape Blue Ltd. Shape Blue Brasil Consultoria
> > Ltda
> > > >> > is a company incorporated in Brasil and is operated under license
> > from
> > > >> > Shape Blue Ltd. ShapeBlue SA Pty Ltd is a company registered
by
> The
> > > >> > Republic of South Africa and is traded under license from Shape
> Blue
> > > >> > Ltd. ShapeBlue is a registered trademark.
> > > >> > This email and any attachments to it may be confidential and
are
> > > >> > intended solely for the use of the individual to whom it is
> > addressed.
> > > >> > Any views or opinions expressed are solely those of the author
and
> > do
> > > >> > not necessarily represent those of Shape Blue Ltd or related
> > > >> > companies. If you are not the intended recipient of this email,
> you
> > > >> > must neither take any action based upon its contents, nor copy
or
> > show
> > > >> > it to anyone. Please contact the sender if you believe you have
> > > received
> > > >> this email in error.
> > > >> >
> > > >> >
> > > >> >
> > > >> >
> > > >> > -----Original Message-----
> > > >> > From: Bharat Kumar [mailto:bharat.kumar@citrix.com]
> > > >> > Sent: Thursday, February 11, 2016 9:57 AM
> > > >> > To: dev@cloudstack.apache.org<mailto:dev@cloudstack.apache.org>
> > > >> > Subject: Re: Important Pending Items
> > > >> >
> > > >> > Hi Sebastien,
> > > >> >
> > > >> > As Raja said, we are actively working on it and we will share
the
> > code
> > > >> > on github soon.
> > > >> >
> > > >> > Thanks,
> > > >> > Bharat.
> > > >> >
> > > >> >
> > > >> > > On 11-Feb-2016, at 2:41 PM, Raja Pullela <
> raja.pullela@citrix.com
> > > >> > <mailto:raja.pullela@citrix.com>> wrote:
> > > >> > >
> > > >> > > Hi Sebastien,
> > > >> > >
> > > >> > > On item (3) - the BVTs have been running consistently (with
same
> > > >> > passrates) and haven't had the time to automate the process of
> > > >> > reporting to an external site.
> > > >> > > Bharat and Sanjeev are working on CI - running BVTs/Regression,
> > > >> > > working
> > > >> > on getting it run automatically and consistently.
> > > >> > > Hope to see it ready soon.
> > > >> > >
> > > >> > > Best,
> > > >> > > Raja
> > > >> > > -----Original Message-----
> > > >> > > From: Sebastien Goasguen [mailto:runseb@gmail.com]
> > > >> > > Sent: Wednesday, February 10, 2016 2:29 PM
> > > >> > > To: dev@cloudstack.apache.org<mailto:dev@cloudstack.apache.org>
> > > >> > > Subject: Important Pending Items
> > > >> > >
> > > >> > > Morning folks,
> > > >> > >
> > > >> > > We have several crucial pending items, that we need to resolve
> > > >> > > before
> > > >> > moving on:
> > > >> > >
> > > >> > > 1- We need an RM for master ( just saw some commits there
that
> > > >> > > should be
> > > >> > reverted or merged properly in other branches).
> > > >> > >
> > > >> > > 2- We need to automate writing release notes, pushing/tagging
> new
> > > >> > > docs
> > > >> > when release come out and announcing releases on website.
> Currently
> > > >> > neither
> > > >> > 4.7 nor 4.8 have been announced.
> > > >> > >
> > > >> > > 3- CI is still almost inexistent
> > > >> > >
> > > >> > > -Sebastien
> > > >> >
> > > >> > Find out more about ShapeBlue and our range of CloudStack related
> > > >> services:
> > > >> > IaaS Cloud Design & Build<
> > > >> > http://shapeblue.com/iaas-cloud-design-and-build//> | CSForge
–
> > rapid
> > > >> > IaaS deployment framework<http://shapeblue.com/csforge/>
> > > >> > CloudStack Consulting<
> http://shapeblue.com/cloudstack-consultancy/>
> > |
> > > >> > CloudStack Software Engineering<
> > > >> > http://shapeblue.com/cloudstack-software-engineering/>
> > > >> > CloudStack Infrastructure Support<
> > > >> > http://shapeblue.com/cloudstack-infrastructure-support/> |
> > CloudStack
> > > >> > Bootcamp Training Courses<
> http://shapeblue.com/cloudstack-training/
> > >
> > > >> >
> > > >> >
> > > >> Find out more about ShapeBlue and our range of CloudStack related
> > > services:
> > > >> IaaS Cloud Design & Build<
> > > >> http://shapeblue.com/iaas-cloud-design-and-build//> | CSForge –
> rapid
> > > >> IaaS deployment framework<http://shapeblue.com/csforge/>
> > > >> CloudStack Consulting<http://shapeblue.com/cloudstack-consultancy/>
> |
> > > >> CloudStack Software Engineering<
> > > >> http://shapeblue.com/cloudstack-software-engineering/>
> > > >> CloudStack Infrastructure Support<
> > > >> http://shapeblue.com/cloudstack-infrastructure-support/> |
> CloudStack
> > > >> Bootcamp Training Courses<http://shapeblue.com/cloudstack-training/
> >
> > > >>
> > >
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message