openwhisk-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Tyson Norris <tnor...@adobe.com.INVALID>
Subject Re: OpenWhisk on Mesos
Date Fri, 14 Jul 2017 17:07:55 GMT
Hi Tim - 
As you mention, running OW in a mixed-workload Mesos cluster can create some issues since
the resource scheduling will be conflicting between Mesos and OW. I think this will be true
of other cluster managers as well, but I’m not sure.

We are experimenting with how to allow OW to operate as a Mesos framework, and delegate container
management to Mesos. There are a number of obstacles currently, such as having no way to provide
an alternate mechanism for launching containers. The simplest version of this would be a system
where a single invoker exists and acts as the Mesos framework, but really needs to support
multiple instances with a shared cache of data to provide high availability. We are working
through some ideas on how to do this, and I believe there is also some similar interest from
others that are using Kubernetes to do something similar. We will be sharing our work on this
with the community asap, but there are a few steps to getting to that point.

Regards,
Tyson 


> On Jul 14, 2017, at 9:08 AM, Boldt, Tim (GE Digital) <Timothy.Boldt@ge.com> wrote:
> 
> Duy,
> 
> Thanks for the updated PR! It contains the fixes I was looking for.
> 
> One of the concerns I have about the current situation is that the config assumes that
OW has carte-blanche permissions in Mesos. For example, both the registrator and invoker talk
directly to the Docker socket, several of the services use HOST networking, and most of the
services reserve dedicated ports instead of using randomly assigned ones. The owners of our
shared Mesos cluster don’t like me doing stuff like this. :-)
> 
> Are most OW-on-Mesos users assuming they will have a dedicated Mesos cluster for OW,
or is the intention to make OW work with a locked-down, shared cluster eventually?
> 
> Tim
> 
> 
> On 7/13/17, 3:49 AM, "Duy Nguyen" <duy.phnguyen@gmail.com> wrote:
> 
>    Hi Tim,
> 
>    Great to see more interest in running OpenWhisk on Mesos.
> 
>    With the deployment structure in the PR (https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fapache%2Finc&data=02%7C01%7C%7Cc61c763eff874cc137a408d4cad29a02%7Cfa7b1b5a7b34438794aed2c178decee1%7C0%7C0%7C636356453251742975&sdata=6SCLSLNPTVUyyZMmAZc9NoYtEpB69wSqQ3I2fU9v7UY%3D&reserved=0
>    ubator-openwhisk-devtools/pull/20), we have our OW instance running on
>    DC/OS. Basically, we wrap each service in a DC/OS package and publish a
>    dedicated universe for it. A deployment means launching those services from
>    the pre-defined packages and apply your apigateway config to set up the
>    server endpoint.
> 
>    Currently one of the main limitations is with the Invoker service: it
>    independently manages resources for running action containers on a single
>    node and does not communicate back to Marathon, hence Marathon is unable to
>    make proper decision on allocating its resources. Some ideas to solve this
>    were discussed in a previous thread [1].
> 
>    I have updated the PR which works well for us with the latest OpenWhisk
>    images. It is still blocked by a Jenkins job for building the
>    "whisk-couchdb" image (currently it's using a manually built couchdb
>    version). When it is ready we should be good to merge.
> 
>    Would love to hear if you have more ideas / suggestions to this approach.
> 
>    Thanks,
>    Duy
> 
>    [1]
>    https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fmail-archives.apache.org%2Fmod_mbox%2Fincubator-openwhisk-dev%2F201705.mbox%2F%253C70F7A242-3BBB-46A5-A49A-61C35C77F68A%40adobe.com%253E&data=02%7C01%7C%7Cc61c763eff874cc137a408d4cad29a02%7Cfa7b1b5a7b34438794aed2c178decee1%7C0%7C0%7C636356453251742975&sdata=p7%2BoWRcbkYftLI6oWlLTl1WVAH4xyhbqywALjgPiLT0%3D&reserved=0
> 
> 
>    On Wed, Jul 12, 2017 at 11:01 PM, Boldt, Tim (GE Digital) <
>    Timothy.Boldt@ge.com> wrote:
> 
>> Is anyone using (or experimenting with) OpenWhisk on Mesos?
>> 
>> I see a 3-month old discussion (https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fapache%2F&data=02%7C01%7C%7Cc61c763eff874cc137a408d4cad29a02%7Cfa7b1b5a7b34438794aed2c178decee1%7C0%7C0%7C636356453251742975&sdata=yrvpqE73Buyn42B5hupK2by07S7GzdOEySmM90c22UM%3D&reserved=0
>> incubator-openwhisk-devtools/issues/17) and a month-old PR (
>> https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fapache%2Fincubator-openwhisk-devtools%2Fpull%2F20&data=02%7C01%7C%7Cc61c763eff874cc137a408d4cad29a02%7Cfa7b1b5a7b34438794aed2c178decee1%7C0%7C0%7C636356453251742975&sdata=Mf0SSiYACYxW2A7nZEHABNYcjYYCuXyjAN0V1u1Yw0U%3D&reserved=0).
However,
>> when I tried it, I had to hack it quite a bit to get it to work for me.
>> 
>> If anyone else has gotten OW working on Mesos, I’d love to hear about your
>> experiences.
>> 
>> 
>> 
> 
> 

Mime
View raw message