cloudstack-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Paul Angus <paul.an...@shapeblue.com>
Subject Re: [PROPOSAL] add native container orchestration service
Date Tue, 07 Feb 2017 07:13:42 GMT
Will is 100% correct.  As I mentioned the Title is misleading.  However, Murali did clarify
in his explanation; this is really about vm cluster orchestration.



________________________________
From: Will Stevens <wstevens@cloudops.com>
Sent: 6 Feb 2017 22:54
To: dev@cloudstack.apache.org
Subject: Re: [PROPOSAL] add native container orchestration service

​My understanding is that what Paul is talking about is what is already
built and IS what the thread is talking about.​

*Will STEVENS*
Lead Developer

<https://goo.gl/NYZ8KK>

On Mon, Feb 6, 2017 at 2:29 PM, Rajesh Ramchandani <
rajesh.ramchandani@accelerite.com> wrote:

> Hi Paul - I think this is different from what the thread was about. The
> conversation was specifically about adding support for container
> orchestrators. You are talking about provisioning a group of VMs. Although,
> this is something I think several Cloudstack users have requested before
> and we should propose a solution to this.
>
> Raj
>
> ________________________________
> From: Paul Angus <paul.angus@shapeblue.com>
> Sent: Monday, February 6, 2017 11:16:41 AM
> To: dev@cloudstack.apache.org
> Subject: RE: [PROPOSAL] add native container orchestration service
>
> #WhatHeSaid
>
> The title is misleading.  The proposal is purely to add the notion of
> Clusters of VMs to CloudStack.  These may be for databases, containers or
> anything else that needs 'clusters' of compute. Self-healing and
> autoscaling are logical next steps to be added.
>
> Those guys at ShapeBlue have open-sourced their whole k8s container
> service piece.  If/when the 'cluster' part of that work is added into
> CloudStack, the k8s specific pieces can be used by anyone who wants to,
> alternatively they could be used for reference in order to create another
> types of cluster.  (or ignored completely).
>
>
>
>
> paul.angus@shapeblue.com
> www.shapeblue.com<http://www.shapeblue.com>
> 53 Chandos Place, Covent Garden, London  WC2N 4HSUK
> @shapeblue
>
>
>
>
> -----Original Message-----
> From: Will Stevens [mailto:williamstevens@gmail.com]
> Sent: 31 January 2017 13:26
> To: dev@cloudstack.apache.org
> Subject: Re: [PROPOSAL] add native container orchestration service
>
> s/cloud-init/cloud-config/
>
> On Jan 31, 2017 7:24 AM, "Will Stevens" <williamstevens@gmail.com> wrote:
>
> > I think that is covered in this proposal. There is nothing k8s
> > specific in this integration (from what I understand), all the k8s
> > details are passed in via the cloud-init configuration after the cluster
> has been provisioned.
> >
> > On Jan 31, 2017 3:06 AM, "Lianghwa Jou" <lianghwa.jou@accelerite.com>
> > wrote:
> >
> >
> > There are many container orchestrators. Those container orchestrators
> > are happy to run on any VMs or bare metal machines. K8s is just one of
> > them and there will be more in the future. It may not be a good idea
> > to make CloudStack to be k8s aware. IMO, the relationship between k8s
> > and cloudstack should be similar to application and os. It probably
> > not a good idea to make your OS to be aware of any specific
> > applications so IMHO I don’t think k8s should be native to CloudStack.
> > It makes more sense to provide some generic services that many
> > applications can take advantages of. Some generic resource grouping
> > service makes sense so a group of VMs, baremetal machines or network
> > can be treated as a single entity. Some life cycle management will be
> > necessary for these entities too. We can deploy k8s, swarm, dcos or
> > even non-container specific services on top of CloudStack. The k8s is
> > changing fast. One single tenant installation may need more than one
> > VM group and may actually requires more (k8s federation). It will be a
> > struggle to be in sync if we try to bring k8s specific knowledge into
> cloudstack.
> >
> > Regards,
> >
> >
> > --
> > Lianghwa Jou
> > VP Engineering, Accelerite
> > email: lianghwa.jou@accelerite.com
> >
> >
> >
> >
> >
> > On 1/29/17, 11:54 PM, "Murali Reddy" <murali.reddy@shapeblue.com> wrote:
> >
> >
> >     I agree with some good views Will has shared and I also agree to
> > the concerns raised by Wido and Eric. IMO we need balance of staying
> > relevant/add new features vs stability of CloudStack and take
> > corrective action if needed. We have two good examples for both. When
> > SDN was hot technology CloudStack ended up with bunch of SDN controller
> integrations.
> > Few years later, now CloudStack is carrying baggage with no
> > maintainers for those plugins, with probably not many of CloudStack
> users needing overlays.
> > On the other hand, other than attempt to liaison with ETSI for NFV no
> > effort was done. OpenStack has become de-facto for NFV. Now for
> > OpenStack, arguably NFV has become larger use case than cloud it self.
> > I concur with Will’s point that with minimal viable solution that does
> > not change the core of CloudStack, and can keep CloudStack relevant is
> > worth to be taken in.
> >
> >     Will,
> >
> >     To your question of how different is from ShapeBlue’s container
> > service, its design, implementation and API semantics etc remain same.
> > ShapeBlue’s container service was true drop in plug-in to CloudStack,
> > with this proposal I am trying to re-work to make it a core CloudStack
> > pluggable service which is part of CloudStack.
> >
> >     Key concepts that this proposal is trying to add
> >
> >         - add notion of ‘container cluster’ as a first class entity in
> > CloudStack. Which is bacially collection of other CloudStack resources
> > (like VM’s, networks, public ip, network rules etc)
> >         - life cycle operation to manage ‘container cluster’ like
> > create, delete, start, stop, scale-up, scale-down, heal etc
> >         - orchestrate container orchestrator control plane on top of
> > provisioned resources
> >
> >     At-least for k8s (which is what this proposal is targeting),
> > integration with k8s is bare minimum. There are cloud-config scripts
> > that automatically setup k8s cluster master and node VM’s. All
> > CloudStack is doing in passing the cloud-config to the core OS VM’s as
> user data.
> >
> >     Regards
> >     Murali Reddy
> >
> >
> >
> >
> >
> >
> >
> >     On 29/01/17, 8:54 AM, "Will Stevens" <williamstevens@gmail.com on
> > behalf of wstevens@cloudops.com> wrote:
> >
> >     >I agree that we need to be careful what we take on and own inside
> >     >CloudStack.  I feel like some of the plugins or integrations
> > which we have
> >     >been "maintaining" may serve us better to abandon, but I feel
> > like that is
> >     >a whole discussion on its own.
> >     >
> >     >In this case, I feel like there is a minimum viable solution
> > which puts
> >     >CloudStack in a pretty good place to enable container orchestration.
> > For
> >     >example, one of the biggest challenges with K8S is the fact that it
> is
> >     >single tenant.  CloudStack has good multi tenancy support and has
> the
> >     >ability to orchestrate the underlying infra quite well.  We will
> > have to be
> >     >very careful not to try to own too deep into the K8S world
> > though, in my
> >     >opinion.  We only want to be responsible for providing the infra
> > (and a way
> >     >to bootstrap K8S ideally) and be able to scale the infra,
> > everything else
> >     >should be owned by the K8S on top.  That is the way I see it
> > anyway, but
> >     >please add your input.
> >     >
> >     >I think it is a liability to try to go too deep, for the same
> > reasons Wido
> >     >and Erik have mentioned.  But I also think we need to take it
> > seriously
> >     >because that train is moving and this may be a good opportunity
> > to stay
> >     >relevant in a rapidly changing market.
> >     >
> >     >*Will STEVENS*
> >     >Lead Developer
> >     >
> >     ><https://goo.gl/NYZ8KK>
> >     >
> >     >On Sat, Jan 28, 2017 at 1:13 PM, Wido den Hollander
> > <wido@widodh.nl>
> > wrote:
> >     >
> >     >>
> >     >> > Op 27 januari 2017 om 16:08 schreef Will Stevens <
> > wstevens@cloudops.com
> >     >> >:
> >     >> >
> >     >> >
> >     >> > Hey Murali,
> >     >> > How different is this proposal than what ShapeBlue already
> > built.  It
> >     >> looks
> >     >> > pretty consistent with the functionality that you guys open
> > sourced in
> >     >> > Seville.
> >     >> >
> >     >> > I have not yet used this functionality, but I have reports
> > that it works
> >     >> > quite well.
> >     >> >
> >     >> > I believe the premise here is to only orchestrate the VM layer
> and
> >     >> > basically expose a "group" of running VMs to the user.  The
> > user is
> >     >> > responsible for configuring K8S or whatever other container
> > orchestrator
> >     >> on
> >     >> > top.  I saw mention of the "cloud-config" scripts in the FS,
> > how are
> >     >> those
> >     >> > exposed to the cluster?  Maybe the FS can expand on that a bit?
> >     >> >
> >     >> > I believe the core feature that is being requested to be
> > added is the
> >     >> > ability to create a group of VMs which will be kept active as
> > a group if
> >     >> at
> >     >> > all possible.  ACS would be responsible for making sure that
> > the number
> >     >> of
> >     >> > VMs specified for the group are in running state and it would
> > spin up new
> >     >> > VMs as needed in order to satisfy the group settings.  In
> > general, it is
> >     >> > understood that any application running on this group would
> > have to be
> >     >> > fault tolerant enough to be able to rediscover a new VM if
> > one fails and
> >     >> is
> >     >> > replaced by a fresh copy.  Is that fair to say?  How is it
> > expected that
> >     >> > this service discovery is done, just by VMs being present on
> > the network?
> >     >> >
> >     >> > As for some of the other people's concerns in this thread.
> >     >> >
> >     >> > - Regarding Wido's remarks.  I understand that there is some
> added
> >     >> > complexity, but I don't feel like the scope of the addition is
> >     >> > unrealistic.  I think the LXC integration was a lot farther
> > out of the
> >     >> > scope of what ACS does then this is.  This does not change
> > the "things"
> >     >> > which ACS orchestrates, it just adds the concept of a
> > grouping of things
> >     >> > which ACS already manages.  I think this is the right
> > approach since it
> >     >> is
> >     >> > not trying to be a container orchestrator.  We will never
> > compete with
> >     >> K8S,
> >     >> > for example, and we should not try, but K8S is here and the
> > market wants
> >     >> > it.  I do think we should be keeping our head up about that
> > fact because
> >     >> > being able to provide a the underlay for K8S is very valuable
> > in the
> >     >> > current marketplace.  I see this functionality as a way to
> > enable K8S
> >     >> > adoption on top of ACS without changing our core values.
> >     >> >
> >     >> > - Regarding Erik's remarks.  The container space is moving
> > fast, but so
> >     >> is
> >     >> > the industry.  If we want to remain relevant, we need to be
> > able to
> >     >> adapt a
> >     >> > bit.  I don't think this is a big shift in what we do, but it
> > is one that
> >     >> > enables people to be able to start running with something
> > like K8S on top
> >     >> > of their existing ACS.  This is something we are interested
> > in doing and
> >     >> so
> >     >> > are our customers.  If we can have a thin layer in ACS which
> > helps enable
> >     >> > the use of K8S (or other container orchestrators) by
> orchestrating
> >     >> > infrastructure, as we already do, and making it easier to adopt
> a
> >     >> container
> >     >> > orchestrator running on top of ACS, I think that gives us a
> > nice foothold
> >     >> > in the market.  I don't really feel it is fair to compare
> > containers to
> >     >> > IPv6.  IPv6 has been out forever and it has taken almost a
> > decade to get
> >     >> > anyone to adopt it.  Containers have really only been here
> > for like 2
> >     >> years
> >     >> > and they are changing the market landscape in a very real way.
> >     >> >
> >     >> > Kind of on topic and kind of off topic.  I think understanding
> our
> >     >> approach
> >     >> > to containers is going to be important for the ACS community
> > as a whole.
> >     >> > If we don't offer that market anything, then we will not be
> > considered
> >     >> and
> >     >> > we will lose market share we can't afford to lose.  If we try
> > to hitch
> >     >> our
> >     >> > horse to that cart too much, we will not be able to be agile
> > enough and
> >     >> > will fail.  I feel like the right approach is for us to know
> > that it is a
> >     >> > thriving market and continue to do what we do, but to extend
> > an olive
> >     >> > branch to that market.  I think this sort of implementation
> > is the right
> >     >> > approach because we are not trying to do too much.  We are
> > simply giving
> >     >> a
> >     >> > foundation on which the next big thing in the container
> > orchestration
> >     >> world
> >     >> > can adopt without us having to compete directly in that
> > space.  I think
> >     >> we
> >     >> > have to focus on what we do best, but at the same time, think
> > about what
> >     >> we
> >     >> > can do to enable that huge market of users to adopt ACS as their
> >     >> > foundation.  The ability to offer VMs and containers in the
> > same data
> >     >> plane
> >     >> > is something we have the ability to do, especially with this
> > approach,
> >     >> and
> >     >> > is something that most other softwares can not do.  The
> > adoption of
> >     >> > containers by bigger organizations will be only part of their
> > workload,
> >     >> > they will still be running VMs for the foreseeable future.
> > Being able to
> >     >> > appeal to that market is going to be important for us.
> >     >> >
> >     >> > Hopefully I don't have too many strong opinions here, but I
> > do think we
> >     >> > need to be thinking about how we move forward in a world which
> is
> >     >> adopting
> >     >> > containers in a very real way.
> >     >> >
> >     >>
> >     >> Understood. I just want to prevent that we add more features to
> > CloudStack
> >     >> which are poorly maintained. Not judging Murali here at all,
> > but I'd rather
> >     >> see CloudStack loose code then have it added.
> >     >>
> >     >> Thinking about LXC, would like to see that removed together
> > with various
> >     >> other network plugins which I think are rarely used.
> >     >>
> >     >> Now, the idea of Murali was misunderstood by me. I think it
> > would be worth
> >     >> more if we would improve our cloud-init support and integration
> > in various
> >     >> tools which makes it much easier to deploy VMs running containers
> ON
> >     >> CloudStack.
> >     >>
> >     >> Not so much changing CloudStack code, but rather tooling around
> it.
> >     >>
> >     >> If we have CloudStack talking to Kubernetes we suddenly have to
> > maintain
> >     >> that API integration. Who's going to do that. Realistically.
> >     >>
> >     >> That's my main worry in this regard.
> >     >>
> >     >> We have so much more work to do in ACS in total that I don't
> > know if this
> >     >> is the realistic route. I talk to many people who are not looking
> at
> >     >> containers and are still working with VMs.
> >     >>
> >     >> There is not a single truth which is true, it really depends on
> > who you
> >     >> ask.
> >     >>
> >     >> Wido
> >     >>
> >     >> > Cheers,
> >     >> >
> >     >> > Will
> >     >> >
> >     >> > *Will STEVENS*
> >     >> > Lead Developer
> >     >> >
> >     >> > <https://goo.gl/NYZ8KK>
> >     >> >
> >     >> > On Fri, Jan 27, 2017 at 5:38 AM, Erik Weber
> > <terbolous@gmail.com>
> > wrote:
> >     >> >
> >     >> > > On Fri, Jan 27, 2017 at 7:20 AM, Murali Reddy <
> > muralimmreddy@gmail.com
> >     >> >
> >     >> > > wrote:
> >     >> > > > All,
> >     >> > > >
> >     >> > > > I would like propose native functionality into CloudStack
> > to provide
> >     >> a
> >     >> > > container service through which users out-of-the box can
> > use to launch
> >     >> > > container based application. Idea is to support ability to
> > orchestrate
> >     >> the
> >     >> > > resources and automate aspects of setting up container
> > orchestrator
> >     >> through
> >     >> > > CloudStack. Public IAAS service providers AWS with its ECS
> > [1] and
> >     >> google
> >     >> > > with GKE [2] already provides ability container applications.
> >     >> Competitive
> >     >> > > cloud orchestration platforms already have native support
> > for container
> >     >> > > service. Users of CloudStack both as public cloud providers
> > and users
> >     >> with
> >     >> > > private clouds will benefit with such functionality.
> >     >> > > >
> >     >> > > > While container orchestrator of user choice can be
> > provisioned on
> >     >> top of
> >     >> > > CloudStack (with out CloudStack being involved) with tools
> like
> >     >> > > TerraForm[3], Ansible[4] etc, advantage of having native
> > orchestration
> >     >> is
> >     >> > > giving user a nice cohesive integration. This proposal
> > would like add a
> >     >> > > notion of first class CloudStack entity called container
> > cluster which
> >     >> can
> >     >> > > be used to provision resources, scale up, scale down, start
> > and stop
> >     >> the
> >     >> > > cluster of VM’s on which containerised applications can
be
> run.
> > For
> >     >> actual
> >     >> > > container orchestration we will still need container
> > orchestrator like
> >     >> > > docker swarm, marathon, kubernetes, but CloudStack
> > container service
> >     >> can
> >     >> > > automate setting up of control place automatically.
> >     >> > > >
> >     >> > >
> >     >> > > To be honest I'm torn on this one.
> >     >> > >
> >     >> > > Containers are a rapid changing thing, and while docker swam,
> >     >> > > kubernetes, rancher or whatnot is popular today, they might
> > not be
> >     >> > > tomorrow.
> >     >> > > They might use CoreOS today, but might not tomorrow.
> >     >> > >
> >     >> > > We have a rather poor track record of staying up to date
> > with new
> >     >> > > features/versions, and adding a feature that is so rapidly
> > changing
> >     >> > > is, I fear, going to be hard to maintain.
> >     >> > > Want an example, look at xenserver. It is one of the most
used
> >     >> > > hypervisors we support, yet it took 6 months or so for us
> > to support
> >     >> > > the latest release.
> >     >> > > Or IPv6...
> >     >> > >
> >     >> > > I don't mean to bash at maintainers/implementers of those
> > features, I
> >     >> > > appreciate all the work being done in every aspect, but I
> > believe we
> >     >> > > should be realistic and realize that we have issues with
> > keeping stuff
> >     >> > > up to date.
> >     >> > >
> >     >> > > I'd say focus on making sure other tools can do their job
> > well against
> >     >> > > CloudStack (kops, rancher, ++), but that does not mean I
> > will
> > -1 the
> >     >> > > idea if anyone really wants to go through with it.
> >     >> > >
> >     >> > > --
> >     >> > > Erik
> >     >> > >
> >     >>
> >
> >     murali.reddy@shapeblue.com
> >     www.shapeblue.com<http://www.shapeblue.com>
> >     53 Chandos Place, Covent Garden, London  WC2N 4HSUK
> >     @shapeblue
> >
> >
> >
> >
> >
> >
> >
> >
> >
> > DISCLAIMER
> > ==========
> > This e-mail may contain privileged and confidential information which
> > is the property of Accelerite, a Persistent Systems business. It is
> > intended only for the use of the individual or entity to which it is
> > addressed. If you are not the intended recipient, you are not
> > authorized to read, retain, copy, print, distribute or use this
> > message. If you have received this communication in error, please
> > notify the sender and delete all copies of this message. Accelerite, a
> > Persistent Systems business does not accept any liability for virus
> infected mails.
> >
> >
> >
>
>
>
> DISCLAIMER
> ==========
> This e-mail may contain privileged and confidential information which is
> the property of Accelerite, a Persistent Systems business. It is intended
> only for the use of the individual or entity to which it is addressed. If
> you are not the intended recipient, you are not authorized to read, retain,
> copy, print, distribute or use this message. If you have received this
> communication in error, please notify the sender and delete all copies of
> this message. Accelerite, a Persistent Systems business does not accept any
> liability for virus infected mails.
>

paul.angus@shapeblue.com 
www.shapeblue.com
53 Chandos Place, Covent Garden, London  WC2N 4HSUK
@shapeblue
  
 

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message