openwhisk-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Markus Thömmes <>
Subject Re: OpenWhisk on Mesos
Date Sat, 15 Jul 2017 18:23:02 GMT
Hi Ben,

this looks great! Any feedback on the container abstractions?

I agree the ContainerPool takes a couple of assumptions as of today, I'd be greatly interested
in reducing those and support using orchestrator APIs just as you're trying to do.

Great work :)


Von meinem iPhone gesendet

> Am 15.07.2017 um 15:50 schrieb Ben Browning <>:
> I'm one of those that desires an alternative implementation to launching
> containers on Kubernetes, as Tyson mentioned. Instead of talking to Docker
> directly we're exploring talking to the Kubernetes API so that resource
> scheduling is still handled by the Kubernetes scheduler and we only have a
> couple of Invoker instances for high availability.
> The route we have right now is an alternative Container implementation (
> that we're prototyping. That implementation is not ready for production use
> by any means and has some serious performance issues we'll need to address.
> But, it works enough that we can explore the tradeoffs when talking
> directly to Docker versus talking to the Kubernetes API and letting
> Kubernetes schedule function runtimes.
> This may or may not be the right approach to take on Kubernetes and Mesos.
> The load-balancing done in the controller and the container pooling done in
> the Invokers today really assume each Invoker manages an independent set of
> Docker containers on separate hardware versus Invokers managing independent
> sets of Docker containers on a shared resource, such as the Kubernetes
> cluster.
> Ben
> On Fri, Jul 14, 2017 at 1:07 PM, Tyson Norris <>
> wrote:
>> Hi Tim -
>> As you mention, running OW in a mixed-workload Mesos cluster can create
>> some issues since the resource scheduling will be conflicting between Mesos
>> and OW. I think this will be true of other cluster managers as well, but
>> I’m not sure.
>> We are experimenting with how to allow OW to operate as a Mesos framework,
>> and delegate container management to Mesos. There are a number of obstacles
>> currently, such as having no way to provide an alternate mechanism for
>> launching containers. The simplest version of this would be a system where
>> a single invoker exists and acts as the Mesos framework, but really needs
>> to support multiple instances with a shared cache of data to provide high
>> availability. We are working through some ideas on how to do this, and I
>> believe there is also some similar interest from others that are using
>> Kubernetes to do something similar. We will be sharing our work on this
>> with the community asap, but there are a few steps to getting to that point.
>> Regards,
>> Tyson
>>> On Jul 14, 2017, at 9:08 AM, Boldt, Tim (GE Digital) <
>>> wrote:
>>> Duy,
>>> Thanks for the updated PR! It contains the fixes I was looking for.
>>> One of the concerns I have about the current situation is that the
>> config assumes that OW has carte-blanche permissions in Mesos. For example,
>> both the registrator and invoker talk directly to the Docker socket,
>> several of the services use HOST networking, and most of the services
>> reserve dedicated ports instead of using randomly assigned ones. The owners
>> of our shared Mesos cluster don’t like me doing stuff like this. :-)
>>> Are most OW-on-Mesos users assuming they will have a dedicated Mesos
>> cluster for OW, or is the intention to make OW work with a locked-down,
>> shared cluster eventually?
>>> Tim
>>> On 7/13/17, 3:49 AM, "Duy Nguyen" <> wrote:
>>>   Hi Tim,
>>>   Great to see more interest in running OpenWhisk on Mesos.
>>>   With the deployment structure in the PR (https://na01.safelinks.
>> 2Fapache%2Finc&data=02%7C01%7C%7Cc61c763eff874cc137a408d4cad29a02%
>> 7Cfa7b1b5a7b34438794aed2c178decee1%7C0%7C0%7C636356453251742975&sdata=
>> 6SCLSLNPTVUyyZMmAZc9NoYtEpB69wSqQ3I2fU9v7UY%3D&reserved=0
>>>   ubator-openwhisk-devtools/pull/20), we have our OW instance running
>> on
>>>   DC/OS. Basically, we wrap each service in a DC/OS package and publish
>> a
>>>   dedicated universe for it. A deployment means launching those
>> services from
>>>   the pre-defined packages and apply your apigateway config to set up
>> the
>>>   server endpoint.
>>>   Currently one of the main limitations is with the Invoker service: it
>>>   independently manages resources for running action containers on a
>> single
>>>   node and does not communicate back to Marathon, hence Marathon is
>> unable to
>>>   make proper decision on allocating its resources. Some ideas to solve
>> this
>>>   were discussed in a previous thread [1].
>>>   I have updated the PR which works well for us with the latest
>> OpenWhisk
>>>   images. It is still blocked by a Jenkins job for building the
>>>   "whisk-couchdb" image (currently it's using a manually built couchdb
>>>   version). When it is ready we should be good to merge.
>>>   Would love to hear if you have more ideas / suggestions to this
>> approach.
>>>   Thanks,
>>>   Duy
>>>   [1]
>> 2Fincubator-openwhisk-dev%2F201705.mbox%2F%253C70F7A242-
>> 7Cc61c763eff874cc137a408d4cad29a02%7Cfa7b1b5a7b34438794aed2c178de
>> cee1%7C0%7C0%7C636356453251742975&sdata=p7%2BoWRcbkYftLI6oWlLTl1WVAH4xyhb
>> qywALjgPiLT0%3D&reserved=0
>>>   On Wed, Jul 12, 2017 at 11:01 PM, Boldt, Tim (GE Digital) <
>>>> wrote:
>>>> Is anyone using (or experimenting with) OpenWhisk on Mesos?
>>>> I see a 3-month old discussion (https://na01.safelinks.
>> 2Fapache%2F&data=02%7C01%7C%7Cc61c763eff874cc137a408d4cad29a02%
>> 7Cfa7b1b5a7b34438794aed2c178decee1%7C0%7C0%7C636356453251742975&sdata=
>> yrvpqE73Buyn42B5hupK2by07S7GzdOEySmM90c22UM%3D&reserved=0
>>>> incubator-openwhisk-devtools/issues/17) and a month-old PR (
>> 2F20&data=02%7C01%7C%7Cc61c763eff874cc137a408d4cad29a02%
>> 7Cfa7b1b5a7b34438794aed2c178decee1%7C0%7C0%7C636356453251742975&sdata=
>> Mf0SSiYACYxW2A7nZEHABNYcjYYCuXyjAN0V1u1Yw0U%3D&reserved=0). However,
>>>> when I tried it, I had to hack it quite a bit to get it to work for me.
>>>> If anyone else has gotten OW working on Mesos, I’d love to hear about
>> your
>>>> experiences.

View raw message