openwhisk-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Markus Thömmes <markusthoem...@me.com>
Subject Re: OpenWhisk on Mesos
Date Sat, 15 Jul 2017 18:23:02 GMT
Hi Ben,

this looks great! Any feedback on the container abstractions?

I agree the ContainerPool takes a couple of assumptions as of today, I'd be greatly interested
in reducing those and support using orchestrator APIs just as you're trying to do.

Great work :)

Cheers,
Markus

Von meinem iPhone gesendet

> Am 15.07.2017 um 15:50 schrieb Ben Browning <bbrownin@redhat.com>:
> 
> I'm one of those that desires an alternative implementation to launching
> containers on Kubernetes, as Tyson mentioned. Instead of talking to Docker
> directly we're exploring talking to the Kubernetes API so that resource
> scheduling is still handled by the Kubernetes scheduler and we only have a
> couple of Invoker instances for high availability.
> 
> The route we have right now is an alternative Container implementation (
> https://github.com/projectodd/incubator-openwhisk/blob/3772c344dbfa421bf361631f96f00f7876eb2adf/core/invoker/src/main/scala/whisk/core/containerpool/kubernetes/KubernetesContainer.scala)
> that we're prototyping. That implementation is not ready for production use
> by any means and has some serious performance issues we'll need to address.
> But, it works enough that we can explore the tradeoffs when talking
> directly to Docker versus talking to the Kubernetes API and letting
> Kubernetes schedule function runtimes.
> 
> This may or may not be the right approach to take on Kubernetes and Mesos.
> The load-balancing done in the controller and the container pooling done in
> the Invokers today really assume each Invoker manages an independent set of
> Docker containers on separate hardware versus Invokers managing independent
> sets of Docker containers on a shared resource, such as the Kubernetes
> cluster.
> 
> Ben
> 
> 
> On Fri, Jul 14, 2017 at 1:07 PM, Tyson Norris <tnorris@adobe.com.invalid>
> wrote:
> 
>> Hi Tim -
>> As you mention, running OW in a mixed-workload Mesos cluster can create
>> some issues since the resource scheduling will be conflicting between Mesos
>> and OW. I think this will be true of other cluster managers as well, but
>> I’m not sure.
>> 
>> We are experimenting with how to allow OW to operate as a Mesos framework,
>> and delegate container management to Mesos. There are a number of obstacles
>> currently, such as having no way to provide an alternate mechanism for
>> launching containers. The simplest version of this would be a system where
>> a single invoker exists and acts as the Mesos framework, but really needs
>> to support multiple instances with a shared cache of data to provide high
>> availability. We are working through some ideas on how to do this, and I
>> believe there is also some similar interest from others that are using
>> Kubernetes to do something similar. We will be sharing our work on this
>> with the community asap, but there are a few steps to getting to that point.
>> 
>> Regards,
>> Tyson
>> 
>> 
>>> On Jul 14, 2017, at 9:08 AM, Boldt, Tim (GE Digital) <
>> Timothy.Boldt@ge.com> wrote:
>>> 
>>> Duy,
>>> 
>>> Thanks for the updated PR! It contains the fixes I was looking for.
>>> 
>>> One of the concerns I have about the current situation is that the
>> config assumes that OW has carte-blanche permissions in Mesos. For example,
>> both the registrator and invoker talk directly to the Docker socket,
>> several of the services use HOST networking, and most of the services
>> reserve dedicated ports instead of using randomly assigned ones. The owners
>> of our shared Mesos cluster don’t like me doing stuff like this. :-)
>>> 
>>> Are most OW-on-Mesos users assuming they will have a dedicated Mesos
>> cluster for OW, or is the intention to make OW work with a locked-down,
>> shared cluster eventually?
>>> 
>>> Tim
>>> 
>>> 
>>> On 7/13/17, 3:49 AM, "Duy Nguyen" <duy.phnguyen@gmail.com> wrote:
>>> 
>>>   Hi Tim,
>>> 
>>>   Great to see more interest in running OpenWhisk on Mesos.
>>> 
>>>   With the deployment structure in the PR (https://na01.safelinks.
>> protection.outlook.com/?url=https%3A%2F%2Fgithub.com%
>> 2Fapache%2Finc&data=02%7C01%7C%7Cc61c763eff874cc137a408d4cad29a02%
>> 7Cfa7b1b5a7b34438794aed2c178decee1%7C0%7C0%7C636356453251742975&sdata=
>> 6SCLSLNPTVUyyZMmAZc9NoYtEpB69wSqQ3I2fU9v7UY%3D&reserved=0
>>>   ubator-openwhisk-devtools/pull/20), we have our OW instance running
>> on
>>>   DC/OS. Basically, we wrap each service in a DC/OS package and publish
>> a
>>>   dedicated universe for it. A deployment means launching those
>> services from
>>>   the pre-defined packages and apply your apigateway config to set up
>> the
>>>   server endpoint.
>>> 
>>>   Currently one of the main limitations is with the Invoker service: it
>>>   independently manages resources for running action containers on a
>> single
>>>   node and does not communicate back to Marathon, hence Marathon is
>> unable to
>>>   make proper decision on allocating its resources. Some ideas to solve
>> this
>>>   were discussed in a previous thread [1].
>>> 
>>>   I have updated the PR which works well for us with the latest
>> OpenWhisk
>>>   images. It is still blocked by a Jenkins job for building the
>>>   "whisk-couchdb" image (currently it's using a manually built couchdb
>>>   version). When it is ready we should be good to merge.
>>> 
>>>   Would love to hear if you have more ideas / suggestions to this
>> approach.
>>> 
>>>   Thanks,
>>>   Duy
>>> 
>>>   [1]
>>>   https://na01.safelinks.protection.outlook.com/?url=
>> https%3A%2F%2Fmail-archives.apache.org%2Fmod_mbox%
>> 2Fincubator-openwhisk-dev%2F201705.mbox%2F%253C70F7A242-
>> 3BBB-46A5-A49A-61C35C77F68A%40adobe.com%253E&data=02%7C01%7C%
>> 7Cc61c763eff874cc137a408d4cad29a02%7Cfa7b1b5a7b34438794aed2c178de
>> cee1%7C0%7C0%7C636356453251742975&sdata=p7%2BoWRcbkYftLI6oWlLTl1WVAH4xyhb
>> qywALjgPiLT0%3D&reserved=0
>>> 
>>> 
>>>   On Wed, Jul 12, 2017 at 11:01 PM, Boldt, Tim (GE Digital) <
>>>   Timothy.Boldt@ge.com> wrote:
>>> 
>>>> Is anyone using (or experimenting with) OpenWhisk on Mesos?
>>>> 
>>>> I see a 3-month old discussion (https://na01.safelinks.
>> protection.outlook.com/?url=https%3A%2F%2Fgithub.com%
>> 2Fapache%2F&data=02%7C01%7C%7Cc61c763eff874cc137a408d4cad29a02%
>> 7Cfa7b1b5a7b34438794aed2c178decee1%7C0%7C0%7C636356453251742975&sdata=
>> yrvpqE73Buyn42B5hupK2by07S7GzdOEySmM90c22UM%3D&reserved=0
>>>> incubator-openwhisk-devtools/issues/17) and a month-old PR (
>>>> https://na01.safelinks.protection.outlook.com/?url=
>> https%3A%2F%2Fgithub.com%2Fapache%2Fincubator-openwhisk-devtools%2Fpull%
>> 2F20&data=02%7C01%7C%7Cc61c763eff874cc137a408d4cad29a02%
>> 7Cfa7b1b5a7b34438794aed2c178decee1%7C0%7C0%7C636356453251742975&sdata=
>> Mf0SSiYACYxW2A7nZEHABNYcjYYCuXyjAN0V1u1Yw0U%3D&reserved=0). However,
>>>> when I tried it, I had to hack it quite a bit to get it to work for me.
>>>> 
>>>> If anyone else has gotten OW working on Mesos, I’d love to hear about
>> your
>>>> experiences.
>>>> 
>>>> 
>>>> 
>>> 
>>> 
>> 
>> 

Mime
View raw message