cloudstack-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Will Stevens <williamstev...@gmail.com>
Subject Re: [PROPOSAL] add native container orchestration service
Date Tue, 31 Jan 2017 12:25:35 GMT
s/cloud-init/cloud-config/

On Jan 31, 2017 7:24 AM, "Will Stevens" <williamstevens@gmail.com> wrote:

> I think that is covered in this proposal. There is nothing k8s specific in
> this integration (from what I understand), all the k8s details are passed
> in via the cloud-init configuration after the cluster has been provisioned.
>
> On Jan 31, 2017 3:06 AM, "Lianghwa Jou" <lianghwa.jou@accelerite.com>
> wrote:
>
>
> There are many container orchestrators. Those container orchestrators are
> happy to run on any VMs or bare metal machines. K8s is just one of them and
> there will be more in the future. It may not be a good idea to make
> CloudStack to be k8s aware. IMO, the relationship between k8s and
> cloudstack should be similar to application and os. It probably not a good
> idea to make your OS to be aware of any specific applications so IMHO I
> don’t think k8s should be native to CloudStack. It makes more sense to
> provide some generic services that many applications can take advantages
> of. Some generic resource grouping service makes sense so a group of VMs,
> baremetal machines or network can be treated as a single entity. Some life
> cycle management will be necessary for these entities too. We can deploy
> k8s, swarm, dcos or even non-container specific services on top of
> CloudStack. The k8s is changing fast. One single tenant installation may
> need more than one VM group and may actually requires more (k8s
> federation). It will be a struggle to be in sync if we try to bring k8s
> specific knowledge into cloudstack.
>
> Regards,
>
>
> --
> Lianghwa Jou
> VP Engineering, Accelerite
> email: lianghwa.jou@accelerite.com
>
>
>
>
>
> On 1/29/17, 11:54 PM, "Murali Reddy" <murali.reddy@shapeblue.com> wrote:
>
>
>     I agree with some good views Will has shared and I also agree to the
> concerns raised by Wido and Eric. IMO we need balance of staying
> relevant/add new features vs stability of CloudStack and take corrective
> action if needed. We have two good examples for both. When SDN was hot
> technology CloudStack ended up with bunch of SDN controller integrations.
> Few years later, now CloudStack is carrying baggage with no maintainers for
> those plugins, with probably not many of CloudStack users needing overlays.
> On the other hand, other than attempt to liaison with ETSI for NFV no
> effort was done. OpenStack has become de-facto for NFV. Now for OpenStack,
> arguably NFV has become larger use case than cloud it self. I concur with
> Will’s point that with minimal viable solution that does not change the
> core of CloudStack, and can keep CloudStack relevant is worth to be taken
> in.
>
>     Will,
>
>     To your question of how different is from ShapeBlue’s container
> service, its design, implementation and API semantics etc remain same.
> ShapeBlue’s container service was true drop in plug-in to CloudStack, with
> this proposal I am trying to re-work to make it a core CloudStack pluggable
> service which is part of CloudStack.
>
>     Key concepts that this proposal is trying to add
>
>         - add notion of ‘container cluster’ as a first class entity in
> CloudStack. Which is bacially collection of other CloudStack resources
> (like VM’s, networks, public ip, network rules etc)
>         - life cycle operation to manage ‘container cluster’ like create,
> delete, start, stop, scale-up, scale-down, heal etc
>         - orchestrate container orchestrator control plane on top of
> provisioned resources
>
>     At-least for k8s (which is what this proposal is targeting),
> integration with k8s is bare minimum. There are cloud-config scripts that
> automatically setup k8s cluster master and node VM’s. All CloudStack is
> doing in passing the cloud-config to the core OS VM’s as user data.
>
>     Regards
>     Murali Reddy
>
>
>
>
>
>
>
>     On 29/01/17, 8:54 AM, "Will Stevens" <williamstevens@gmail.com on
> behalf of wstevens@cloudops.com> wrote:
>
>     >I agree that we need to be careful what we take on and own inside
>     >CloudStack.  I feel like some of the plugins or integrations which we
> have
>     >been "maintaining" may serve us better to abandon, but I feel like
> that is
>     >a whole discussion on its own.
>     >
>     >In this case, I feel like there is a minimum viable solution which
> puts
>     >CloudStack in a pretty good place to enable container orchestration.
> For
>     >example, one of the biggest challenges with K8S is the fact that it is
>     >single tenant.  CloudStack has good multi tenancy support and has the
>     >ability to orchestrate the underlying infra quite well.  We will have
> to be
>     >very careful not to try to own too deep into the K8S world though, in
> my
>     >opinion.  We only want to be responsible for providing the infra (and
> a way
>     >to bootstrap K8S ideally) and be able to scale the infra, everything
> else
>     >should be owned by the K8S on top.  That is the way I see it anyway,
> but
>     >please add your input.
>     >
>     >I think it is a liability to try to go too deep, for the same reasons
> Wido
>     >and Erik have mentioned.  But I also think we need to take it
> seriously
>     >because that train is moving and this may be a good opportunity to
> stay
>     >relevant in a rapidly changing market.
>     >
>     >*Will STEVENS*
>     >Lead Developer
>     >
>     ><https://goo.gl/NYZ8KK>
>     >
>     >On Sat, Jan 28, 2017 at 1:13 PM, Wido den Hollander <wido@widodh.nl>
> wrote:
>     >
>     >>
>     >> > Op 27 januari 2017 om 16:08 schreef Will Stevens <
> wstevens@cloudops.com
>     >> >:
>     >> >
>     >> >
>     >> > Hey Murali,
>     >> > How different is this proposal than what ShapeBlue already
> built.  It
>     >> looks
>     >> > pretty consistent with the functionality that you guys open
> sourced in
>     >> > Seville.
>     >> >
>     >> > I have not yet used this functionality, but I have reports that
> it works
>     >> > quite well.
>     >> >
>     >> > I believe the premise here is to only orchestrate the VM layer and
>     >> > basically expose a "group" of running VMs to the user.  The user
> is
>     >> > responsible for configuring K8S or whatever other container
> orchestrator
>     >> on
>     >> > top.  I saw mention of the "cloud-config" scripts in the FS, how
> are
>     >> those
>     >> > exposed to the cluster?  Maybe the FS can expand on that a bit?
>     >> >
>     >> > I believe the core feature that is being requested to be added is
> the
>     >> > ability to create a group of VMs which will be kept active as a
> group if
>     >> at
>     >> > all possible.  ACS would be responsible for making sure that the
> number
>     >> of
>     >> > VMs specified for the group are in running state and it would
> spin up new
>     >> > VMs as needed in order to satisfy the group settings.  In
> general, it is
>     >> > understood that any application running on this group would have
> to be
>     >> > fault tolerant enough to be able to rediscover a new VM if one
> fails and
>     >> is
>     >> > replaced by a fresh copy.  Is that fair to say?  How is it
> expected that
>     >> > this service discovery is done, just by VMs being present on the
> network?
>     >> >
>     >> > As for some of the other people's concerns in this thread.
>     >> >
>     >> > - Regarding Wido's remarks.  I understand that there is some added
>     >> > complexity, but I don't feel like the scope of the addition is
>     >> > unrealistic.  I think the LXC integration was a lot farther out
> of the
>     >> > scope of what ACS does then this is.  This does not change the
> "things"
>     >> > which ACS orchestrates, it just adds the concept of a grouping of
> things
>     >> > which ACS already manages.  I think this is the right approach
> since it
>     >> is
>     >> > not trying to be a container orchestrator.  We will never compete
> with
>     >> K8S,
>     >> > for example, and we should not try, but K8S is here and the
> market wants
>     >> > it.  I do think we should be keeping our head up about that fact
> because
>     >> > being able to provide a the underlay for K8S is very valuable in
> the
>     >> > current marketplace.  I see this functionality as a way to enable
> K8S
>     >> > adoption on top of ACS without changing our core values.
>     >> >
>     >> > - Regarding Erik's remarks.  The container space is moving fast,
> but so
>     >> is
>     >> > the industry.  If we want to remain relevant, we need to be able
> to
>     >> adapt a
>     >> > bit.  I don't think this is a big shift in what we do, but it is
> one that
>     >> > enables people to be able to start running with something like
> K8S on top
>     >> > of their existing ACS.  This is something we are interested in
> doing and
>     >> so
>     >> > are our customers.  If we can have a thin layer in ACS which
> helps enable
>     >> > the use of K8S (or other container orchestrators) by orchestrating
>     >> > infrastructure, as we already do, and making it easier to adopt a
>     >> container
>     >> > orchestrator running on top of ACS, I think that gives us a nice
> foothold
>     >> > in the market.  I don't really feel it is fair to compare
> containers to
>     >> > IPv6.  IPv6 has been out forever and it has taken almost a decade
> to get
>     >> > anyone to adopt it.  Containers have really only been here for
> like 2
>     >> years
>     >> > and they are changing the market landscape in a very real way.
>     >> >
>     >> > Kind of on topic and kind of off topic.  I think understanding our
>     >> approach
>     >> > to containers is going to be important for the ACS community as a
> whole.
>     >> > If we don't offer that market anything, then we will not be
> considered
>     >> and
>     >> > we will lose market share we can't afford to lose.  If we try to
> hitch
>     >> our
>     >> > horse to that cart too much, we will not be able to be agile
> enough and
>     >> > will fail.  I feel like the right approach is for us to know that
> it is a
>     >> > thriving market and continue to do what we do, but to extend an
> olive
>     >> > branch to that market.  I think this sort of implementation is
> the right
>     >> > approach because we are not trying to do too much.  We are simply
> giving
>     >> a
>     >> > foundation on which the next big thing in the container
> orchestration
>     >> world
>     >> > can adopt without us having to compete directly in that space.  I
> think
>     >> we
>     >> > have to focus on what we do best, but at the same time, think
> about what
>     >> we
>     >> > can do to enable that huge market of users to adopt ACS as their
>     >> > foundation.  The ability to offer VMs and containers in the same
> data
>     >> plane
>     >> > is something we have the ability to do, especially with this
> approach,
>     >> and
>     >> > is something that most other softwares can not do.  The adoption
> of
>     >> > containers by bigger organizations will be only part of their
> workload,
>     >> > they will still be running VMs for the foreseeable future. Being
> able to
>     >> > appeal to that market is going to be important for us.
>     >> >
>     >> > Hopefully I don't have too many strong opinions here, but I do
> think we
>     >> > need to be thinking about how we move forward in a world which is
>     >> adopting
>     >> > containers in a very real way.
>     >> >
>     >>
>     >> Understood. I just want to prevent that we add more features to
> CloudStack
>     >> which are poorly maintained. Not judging Murali here at all, but
> I'd rather
>     >> see CloudStack loose code then have it added.
>     >>
>     >> Thinking about LXC, would like to see that removed together with
> various
>     >> other network plugins which I think are rarely used.
>     >>
>     >> Now, the idea of Murali was misunderstood by me. I think it would
> be worth
>     >> more if we would improve our cloud-init support and integration in
> various
>     >> tools which makes it much easier to deploy VMs running containers ON
>     >> CloudStack.
>     >>
>     >> Not so much changing CloudStack code, but rather tooling around it.
>     >>
>     >> If we have CloudStack talking to Kubernetes we suddenly have to
> maintain
>     >> that API integration. Who's going to do that. Realistically.
>     >>
>     >> That's my main worry in this regard.
>     >>
>     >> We have so much more work to do in ACS in total that I don't know
> if this
>     >> is the realistic route. I talk to many people who are not looking at
>     >> containers and are still working with VMs.
>     >>
>     >> There is not a single truth which is true, it really depends on who
> you
>     >> ask.
>     >>
>     >> Wido
>     >>
>     >> > Cheers,
>     >> >
>     >> > Will
>     >> >
>     >> > *Will STEVENS*
>     >> > Lead Developer
>     >> >
>     >> > <https://goo.gl/NYZ8KK>
>     >> >
>     >> > On Fri, Jan 27, 2017 at 5:38 AM, Erik Weber <terbolous@gmail.com>
> wrote:
>     >> >
>     >> > > On Fri, Jan 27, 2017 at 7:20 AM, Murali Reddy <
> muralimmreddy@gmail.com
>     >> >
>     >> > > wrote:
>     >> > > > All,
>     >> > > >
>     >> > > > I would like propose native functionality into CloudStack
to
> provide
>     >> a
>     >> > > container service through which users out-of-the box can use to
> launch
>     >> > > container based application. Idea is to support ability to
> orchestrate
>     >> the
>     >> > > resources and automate aspects of setting up container
> orchestrator
>     >> through
>     >> > > CloudStack. Public IAAS service providers AWS with its ECS [1]
> and
>     >> google
>     >> > > with GKE [2] already provides ability container applications.
>     >> Competitive
>     >> > > cloud orchestration platforms already have native support for
> container
>     >> > > service. Users of CloudStack both as public cloud providers and
> users
>     >> with
>     >> > > private clouds will benefit with such functionality.
>     >> > > >
>     >> > > > While container orchestrator of user choice can be
> provisioned on
>     >> top of
>     >> > > CloudStack (with out CloudStack being involved) with tools like
>     >> > > TerraForm[3], Ansible[4] etc, advantage of having native
> orchestration
>     >> is
>     >> > > giving user a nice cohesive integration. This proposal would
> like add a
>     >> > > notion of first class CloudStack entity called container
> cluster which
>     >> can
>     >> > > be used to provision resources, scale up, scale down, start and
> stop
>     >> the
>     >> > > cluster of VM’s on which containerised applications can be run.
> For
>     >> actual
>     >> > > container orchestration we will still need container
> orchestrator like
>     >> > > docker swarm, marathon, kubernetes, but CloudStack container
> service
>     >> can
>     >> > > automate setting up of control place automatically.
>     >> > > >
>     >> > >
>     >> > > To be honest I'm torn on this one.
>     >> > >
>     >> > > Containers are a rapid changing thing, and while docker swam,
>     >> > > kubernetes, rancher or whatnot is popular today, they might not
> be
>     >> > > tomorrow.
>     >> > > They might use CoreOS today, but might not tomorrow.
>     >> > >
>     >> > > We have a rather poor track record of staying up to date with
> new
>     >> > > features/versions, and adding a feature that is so rapidly
> changing
>     >> > > is, I fear, going to be hard to maintain.
>     >> > > Want an example, look at xenserver. It is one of the most used
>     >> > > hypervisors we support, yet it took 6 months or so for us to
> support
>     >> > > the latest release.
>     >> > > Or IPv6...
>     >> > >
>     >> > > I don't mean to bash at maintainers/implementers of those
> features, I
>     >> > > appreciate all the work being done in every aspect, but I
> believe we
>     >> > > should be realistic and realize that we have issues with
> keeping stuff
>     >> > > up to date.
>     >> > >
>     >> > > I'd say focus on making sure other tools can do their job well
> against
>     >> > > CloudStack (kops, rancher, ++), but that does not mean I will
> -1 the
>     >> > > idea if anyone really wants to go through with it.
>     >> > >
>     >> > > --
>     >> > > Erik
>     >> > >
>     >>
>
>     murali.reddy@shapeblue.com
>     www.shapeblue.com
>     53 Chandos Place, Covent Garden, London  WC2N 4HSUK
>     @shapeblue
>
>
>
>
>
>
>
>
>
> DISCLAIMER
> ==========
> This e-mail may contain privileged and confidential information which is
> the property of Accelerite, a Persistent Systems business. It is intended
> only for the use of the individual or entity to which it is addressed. If
> you are not the intended recipient, you are not authorized to read, retain,
> copy, print, distribute or use this message. If you have received this
> communication in error, please notify the sender and delete all copies of
> this message. Accelerite, a Persistent Systems business does not accept any
> liability for virus infected mails.
>
>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message