openwhisk-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ben Browning <>
Subject Re: Mesos integration
Date Wed, 31 May 2017 21:41:50 GMT
I believe a lot of Tyson's observations around Mesos integration also
apply to Kubernetes integration. For example, I've been experimenting
with an approach that replaces the DockerContainer with a
KubernetesContainer. The KubernetesContainer also ignores the pause
and resume of functions and also deploys containers to arbitrary nodes
in the Kubernetes cluster.

Here's a rough but working (except for log shipping) prototype of that

So, I'd like to add a +1 Tyson's notes around isolating docker client
usage and log collection. I believe they both equally apply to a
Kubernetes environment as well. And, more broadly, we'll need to
figure out how to make the various Container implementations pluggable
(or at least configurable) for Docker, Kubernetes, & Mesos.


On Wed, May 31, 2017 at 5:24 PM, Carlos Santana <> wrote:
> Hi Tyson
> Thanks for reaching out, having Mesos integration in OpenWhisk is something
> I have heard some users asking about and any help to figure out how to best
> integrate are very welcome.
> As you already aware OpenWhisk Invoker is a critical path component
> optimized for performance, to get those container pause and unpause in
> single digit milliseconds that's the reason for the recent change from
> using docker engine down to runc
> Here is my initial notes, take into account I don't have first hand
> experience with Mesos, which is why is good to have folks like you join :-)
> I think extending in a way that a different mode can be use would be useful
> to plug a different orchestration methodology.
> For log collection we are looking into sending logs into ELK (Elastic
> search) there is some early  investigation going on, so I see after this
> integration, mesos can be told to which ELK to send the logs, and the
> Invoker to get the results.
> In terms of scala vs. nodejs, I would recommend to use scala with the java
> client, since there could be some components being integrated into Invoker
> and code sharing.
> How you want to follow up, maybe you have something to show us or maybe
> setup a hangout call to meet
> I think Dragos from Adobe, might have a lot more feedback than me since I
> think he has first hand experience
> --Carlos
> On Sat, May 27, 2017 at 2:44 PM Tyson Norris <>
> wrote:
>> Hi All -
>> I’ve been considering how we can leverage mesos resource scheduling within
>> OpenWhisk, and wanted to share some ideas and solicit some feedback.
>> In general, I’ve been experimenting with an approach that replaces the
>> DockerContainer implementation with a MesosTask. The MesosTask will have
>> effectively the same lifecycle (ignoring pause/resume functions) and the
>> same interface (HTTP api to the container), the main difference being that
>> the container will be deployed to some arbitrary host in the cluster.
>> There are a few broad topics around this, including:
>> - docker client usage needs to be better isolated (e.g. cleaning up
>> existing containers, reconciling containers already running at time of
>> invoker startup, etc). I think this will be mostly straightforward.
>> - log collection - I’m not sure the best approach here, but one is to
>> completely decouple log collection from activation execution, and then
>> provide a mesos-specific impl
>> - container state tracking + load balancing - obviously there is potential
>> for conflict if 2 invokers schedule activations on the same container in
>> the cluster (since the state tracking of the container would still be in
>> the invoker). This would imply some extension to the ContainerPool as well.
>> - mesos framework - we’ve discussed internally a bit, and some preferences
>> I have are: leverage the mesos http api (avoid the older jni libs if
>> possible) and provide an independent framework application which provides a
>> (simpler)  http api that is consumed by the invoker (if mesos integration
>> is enabled in the deployment). This way the framework deployment can be
>> isolated from controller/invoker deployment, and interaction with docker
>> containers is mostly the same (except for logging).
>> - There are a few options of implementing mesos http api clients like:
>> rxjava client, and nodejs client, but I've not seen any good Scala client
>> to date, so we may either provide a scala app using the rxjava client, or
>> create.a new client in Scala.
>> Let me know if you have any thoughts around any of these, and I will share
>> more details as they come.
>> Thanks
>> Tyson

View raw message