openwhisk-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ben Browning <bbrow...@redhat.com>
Subject Re: OpenWhisk on Mesos
Date Sat, 15 Jul 2017 13:50:20 GMT
I'm one of those that desires an alternative implementation to launching
containers on Kubernetes, as Tyson mentioned. Instead of talking to Docker
directly we're exploring talking to the Kubernetes API so that resource
scheduling is still handled by the Kubernetes scheduler and we only have a
couple of Invoker instances for high availability.

The route we have right now is an alternative Container implementation (
https://github.com/projectodd/incubator-openwhisk/blob/3772c344dbfa421bf361631f96f00f7876eb2adf/core/invoker/src/main/scala/whisk/core/containerpool/kubernetes/KubernetesContainer.scala)
that we're prototyping. That implementation is not ready for production use
by any means and has some serious performance issues we'll need to address.
But, it works enough that we can explore the tradeoffs when talking
directly to Docker versus talking to the Kubernetes API and letting
Kubernetes schedule function runtimes.

This may or may not be the right approach to take on Kubernetes and Mesos.
The load-balancing done in the controller and the container pooling done in
the Invokers today really assume each Invoker manages an independent set of
Docker containers on separate hardware versus Invokers managing independent
sets of Docker containers on a shared resource, such as the Kubernetes
cluster.

Ben


On Fri, Jul 14, 2017 at 1:07 PM, Tyson Norris <tnorris@adobe.com.invalid>
wrote:

> Hi Tim -
> As you mention, running OW in a mixed-workload Mesos cluster can create
> some issues since the resource scheduling will be conflicting between Mesos
> and OW. I think this will be true of other cluster managers as well, but
> I’m not sure.
>
> We are experimenting with how to allow OW to operate as a Mesos framework,
> and delegate container management to Mesos. There are a number of obstacles
> currently, such as having no way to provide an alternate mechanism for
> launching containers. The simplest version of this would be a system where
> a single invoker exists and acts as the Mesos framework, but really needs
> to support multiple instances with a shared cache of data to provide high
> availability. We are working through some ideas on how to do this, and I
> believe there is also some similar interest from others that are using
> Kubernetes to do something similar. We will be sharing our work on this
> with the community asap, but there are a few steps to getting to that point.
>
> Regards,
> Tyson
>
>
> > On Jul 14, 2017, at 9:08 AM, Boldt, Tim (GE Digital) <
> Timothy.Boldt@ge.com> wrote:
> >
> > Duy,
> >
> > Thanks for the updated PR! It contains the fixes I was looking for.
> >
> > One of the concerns I have about the current situation is that the
> config assumes that OW has carte-blanche permissions in Mesos. For example,
> both the registrator and invoker talk directly to the Docker socket,
> several of the services use HOST networking, and most of the services
> reserve dedicated ports instead of using randomly assigned ones. The owners
> of our shared Mesos cluster don’t like me doing stuff like this. :-)
> >
> > Are most OW-on-Mesos users assuming they will have a dedicated Mesos
> cluster for OW, or is the intention to make OW work with a locked-down,
> shared cluster eventually?
> >
> > Tim
> >
> >
> > On 7/13/17, 3:49 AM, "Duy Nguyen" <duy.phnguyen@gmail.com> wrote:
> >
> >    Hi Tim,
> >
> >    Great to see more interest in running OpenWhisk on Mesos.
> >
> >    With the deployment structure in the PR (https://na01.safelinks.
> protection.outlook.com/?url=https%3A%2F%2Fgithub.com%
> 2Fapache%2Finc&data=02%7C01%7C%7Cc61c763eff874cc137a408d4cad29a02%
> 7Cfa7b1b5a7b34438794aed2c178decee1%7C0%7C0%7C636356453251742975&sdata=
> 6SCLSLNPTVUyyZMmAZc9NoYtEpB69wSqQ3I2fU9v7UY%3D&reserved=0
> >    ubator-openwhisk-devtools/pull/20), we have our OW instance running
> on
> >    DC/OS. Basically, we wrap each service in a DC/OS package and publish
> a
> >    dedicated universe for it. A deployment means launching those
> services from
> >    the pre-defined packages and apply your apigateway config to set up
> the
> >    server endpoint.
> >
> >    Currently one of the main limitations is with the Invoker service: it
> >    independently manages resources for running action containers on a
> single
> >    node and does not communicate back to Marathon, hence Marathon is
> unable to
> >    make proper decision on allocating its resources. Some ideas to solve
> this
> >    were discussed in a previous thread [1].
> >
> >    I have updated the PR which works well for us with the latest
> OpenWhisk
> >    images. It is still blocked by a Jenkins job for building the
> >    "whisk-couchdb" image (currently it's using a manually built couchdb
> >    version). When it is ready we should be good to merge.
> >
> >    Would love to hear if you have more ideas / suggestions to this
> approach.
> >
> >    Thanks,
> >    Duy
> >
> >    [1]
> >    https://na01.safelinks.protection.outlook.com/?url=
> https%3A%2F%2Fmail-archives.apache.org%2Fmod_mbox%
> 2Fincubator-openwhisk-dev%2F201705.mbox%2F%253C70F7A242-
> 3BBB-46A5-A49A-61C35C77F68A%40adobe.com%253E&data=02%7C01%7C%
> 7Cc61c763eff874cc137a408d4cad29a02%7Cfa7b1b5a7b34438794aed2c178de
> cee1%7C0%7C0%7C636356453251742975&sdata=p7%2BoWRcbkYftLI6oWlLTl1WVAH4xyhb
> qywALjgPiLT0%3D&reserved=0
> >
> >
> >    On Wed, Jul 12, 2017 at 11:01 PM, Boldt, Tim (GE Digital) <
> >    Timothy.Boldt@ge.com> wrote:
> >
> >> Is anyone using (or experimenting with) OpenWhisk on Mesos?
> >>
> >> I see a 3-month old discussion (https://na01.safelinks.
> protection.outlook.com/?url=https%3A%2F%2Fgithub.com%
> 2Fapache%2F&data=02%7C01%7C%7Cc61c763eff874cc137a408d4cad29a02%
> 7Cfa7b1b5a7b34438794aed2c178decee1%7C0%7C0%7C636356453251742975&sdata=
> yrvpqE73Buyn42B5hupK2by07S7GzdOEySmM90c22UM%3D&reserved=0
> >> incubator-openwhisk-devtools/issues/17) and a month-old PR (
> >> https://na01.safelinks.protection.outlook.com/?url=
> https%3A%2F%2Fgithub.com%2Fapache%2Fincubator-openwhisk-devtools%2Fpull%
> 2F20&data=02%7C01%7C%7Cc61c763eff874cc137a408d4cad29a02%
> 7Cfa7b1b5a7b34438794aed2c178decee1%7C0%7C0%7C636356453251742975&sdata=
> Mf0SSiYACYxW2A7nZEHABNYcjYYCuXyjAN0V1u1Yw0U%3D&reserved=0). However,
> >> when I tried it, I had to hack it quite a bit to get it to work for me.
> >>
> >> If anyone else has gotten OW working on Mesos, I’d love to hear about
> your
> >> experiences.
> >>
> >>
> >>
> >
> >
>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message