openwhisk-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Tyson Norris <>
Subject Re: Mesos integration
Date Thu, 01 Jun 2017 00:02:33 GMT

One thing I think will help is isolating the http contract with containers from the management
(start/stop etc) of the container. (Currently the http api client is built into the DockerContainer

After that, providing an interface for a configured container management would be able to
dictate the approach to use in a particular env. Note that this would not preclude using the
“docker” container management within a mesos or kubernetes env, if people wanted to still
do that (there will be tradeoffs, like start/stop time vs cluster-level pool visibility).

RE: mesos details:I have some concerns with using existing java libs from Scala - the old
libs all use, which can be a nuisance to deal with, and the newer libs rely on
some java method overloading which brings up some compiler issues in Scala. I haven’t exhaustively
looked at it, but currently I’m testing out the mesos-rxjava lib. If anyone has opinions
or experience with this lib, let me know. (I work with Dragos BTW, so will be soliciting feedback
from him regularly)


> On May 31, 2017, at 3:19 PM, Carlos Santana <> wrote:
> Yeah I agree between Mesos and Kubernetes work there should be some common
> integration extension points that would serve both use cases to some extent
> On Wed, May 31, 2017 at 5:42 PM Ben Browning <> wrote:
>> I believe a lot of Tyson's observations around Mesos integration also
>> apply to Kubernetes integration. For example, I've been experimenting
>> with an approach that replaces the DockerContainer with a
>> KubernetesContainer. The KubernetesContainer also ignores the pause
>> and resume of functions and also deploys containers to arbitrary nodes
>> in the Kubernetes cluster.
>> Here's a rough but working (except for log shipping) prototype of that
>> -
>> So, I'd like to add a +1 Tyson's notes around isolating docker client
>> usage and log collection. I believe they both equally apply to a
>> Kubernetes environment as well. And, more broadly, we'll need to
>> figure out how to make the various Container implementations pluggable
>> (or at least configurable) for Docker, Kubernetes, & Mesos.
>> Ben
>> On Wed, May 31, 2017 at 5:24 PM, Carlos Santana <>
>> wrote:
>>> Hi Tyson
>>> Thanks for reaching out, having Mesos integration in OpenWhisk is
>> something
>>> I have heard some users asking about and any help to figure out how to
>> best
>>> integrate are very welcome.
>>> As you already aware OpenWhisk Invoker is a critical path component
>>> optimized for performance, to get those container pause and unpause in
>>> single digit milliseconds that's the reason for the recent change from
>>> using docker engine down to runc
>>> Here is my initial notes, take into account I don't have first hand
>>> experience with Mesos, which is why is good to have folks like you join
>> :-)
>>> I think extending in a way that a different mode can be use would be
>> useful
>>> to plug a different orchestration methodology.
>>> For log collection we are looking into sending logs into ELK (Elastic
>>> search) there is some early  investigation going on, so I see after this
>>> integration, mesos can be told to which ELK to send the logs, and the
>>> Invoker to get the results.
>>> In terms of scala vs. nodejs, I would recommend to use scala with the
>> java
>>> client, since there could be some components being integrated into
>> Invoker
>>> and code sharing.
>>> How you want to follow up, maybe you have something to show us or maybe
>>> setup a hangout call to meet
>>> I think Dragos from Adobe, might have a lot more feedback than me since I
>>> think he has first hand experience
>>> --Carlos
>>> On Sat, May 27, 2017 at 2:44 PM Tyson Norris <>
>>> wrote:
>>>> Hi All -
>>>> I’ve been considering how we can leverage mesos resource scheduling
>> within
>>>> OpenWhisk, and wanted to share some ideas and solicit some feedback.
>>>> In general, I’ve been experimenting with an approach that replaces the
>>>> DockerContainer implementation with a MesosTask. The MesosTask will have
>>>> effectively the same lifecycle (ignoring pause/resume functions) and the
>>>> same interface (HTTP api to the container), the main difference being
>> that
>>>> the container will be deployed to some arbitrary host in the cluster.
>>>> There are a few broad topics around this, including:
>>>> - docker client usage needs to be better isolated (e.g. cleaning up
>>>> existing containers, reconciling containers already running at time of
>>>> invoker startup, etc). I think this will be mostly straightforward.
>>>> - log collection - I’m not sure the best approach here, but one is to
>>>> completely decouple log collection from activation execution, and then
>>>> provide a mesos-specific impl
>>>> - container state tracking + load balancing - obviously there is
>> potential
>>>> for conflict if 2 invokers schedule activations on the same container in
>>>> the cluster (since the state tracking of the container would still be in
>>>> the invoker). This would imply some extension to the ContainerPool as
>> well.
>>>> - mesos framework - we’ve discussed internally a bit, and some
>> preferences
>>>> I have are: leverage the mesos http api (avoid the older jni libs if
>>>> possible) and provide an independent framework application which
>> provides a
>>>> (simpler)  http api that is consumed by the invoker (if mesos
>> integration
>>>> is enabled in the deployment). This way the framework deployment can be
>>>> isolated from controller/invoker deployment, and interaction with docker
>>>> containers is mostly the same (except for logging).
>>>> - There are a few options of implementing mesos http api clients like:
>>>> rxjava client, and nodejs client, but I've not seen any good Scala
>> client
>>>> to date, so we may either provide a scala app using the rxjava client,
>> or
>>>> create.a new client in Scala.
>>>> Let me know if you have any thoughts around any of these, and I will
>> share
>>>> more details as they come.
>>>> Thanks
>>>> Tyson

View raw message