Mailing-List: contact dev-help@aurora.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@aurora.apache.org
MIME-Version: 1.0
In-Reply-To: 
 <CAAATh-aXM8OL+ERwK4mA4_pv+rhMz=ERDutK1XJLcu2GhxpD5g@mail.gmail.com>
References: 
 <CAAATh-aXM8OL+ERwK4mA4_pv+rhMz=ERDutK1XJLcu2GhxpD5g@mail.gmail.com>
Date: Fri, 14 Aug 2015 12:17:47 -0700
Message-ID: 
 <CAL=oppdHWonLTEiksz76MK9CWUPEm1MfSb4M5FYiar84gegsXw@mail.gmail.com>
Subject: Re: Improvements to container support with custom executor
From: Jie Yu <yujie.jay@gmail.com>
To: dev@aurora.apache.org
Cc: "user@mesos.apache.org" <user@mesos.apache.org>,
 Tim Chen <tim@mesosphere.io>
Content-Type: multipart/alternative; boundary=bcaec51b9dbb9aecfd051d4a4e39

--bcaec51b9dbb9aecfd051d4a4e39
Content-Type: text/plain; charset=UTF-8

Kevin,

Thanks for bringing this up! This question is related to the work we are
currently doing for Filesystem Isolation (MESOS-2386
<https://issues.apache.org/jira/browse/MESOS-2386>) and Unified Container (
MESOS-2840 <https://issues.apache.org/jira/browse/MESOS-2840>).

In fact, if you think about using Mesos containerizer and want to run a
task (using command executor) under a specified image, we have the same
issue for Mesos' own command executor (MESOS-3004
<https://issues.apache.org/jira/browse/MESOS-3004>).

I've proposed a solution in this doc
<https://docs.google.com/document/d/16hyLVRL0nz-KBts1J5stGyxZPniFPbPbs7R-ZRQVCH4/edit?usp=sharing>.
The main idea is that we allow the custom executor to still run under the
host filesystem (so that it does not have to deal with dependency issues).
Mesos will provision the image (the image requested by the user), and mount
in as a Volume in the container. The custom executor is responsible for
chrooting (or pivot_root if it has root privilege) into the user specified
image before execing the user process.

The above proposal is for only Mesos containerizer currently. But with the
Unified Container (MESOS-2840
<https://issues.apache.org/jira/browse/MESOS-2840>) work, Mesos
containerizer is going to support Docker images as well.

cc Tim Chen

- Jie


On Fri, Aug 14, 2015 at 11:39 AM, Kevin Sweeney <
ksweeney@twitter.com.invalid> wrote:

> (cross-posting for a wider audience)
>
> Hi folks,
>
> With mesos-0.23.0 it looks like a new dependency made it in for TLS
> support. While this is fine in theory it actually makes the Docker
> Containerizer story very difficult to reason about.
>
> Here's the situation: Aurora uses a custom Python executor for its tasks.
> Because of the design of the containerizer the executor runs in the context
> of the container. This means that the container has to be able to run the
> executor (in practice this means it needs a python2.7 installation and some
> shared libraries libmesos links to). Since most containers don't contain
> Aurora's executor, Aurora hacked around this by using the executor in
> $MESOS_SANDBOX plus a requirement that hosted containers be able to run the
> executor. However, with the upgrade to mesos-0.23.0 the containers that
> could run the 0.22.0 executor no longer work due to the new dependency on
> libcurl-nss.
>
> This is not a problem limited to Docker - I don't see how this design will
> work with *any* container runtime - we can never upgrade the executor
> without upgrading all the containers to contain its new dependencies, which
> at a minimum means we must rebuild them whenever mesos gains a new
> dependency.
>
> Does anyone with more experience with these APIs have a suggestion here? It
> seems we need to make the executor run in the context of the host OS and
> aware of the container (or maybe we have mesos launch a container with the
> executor+its dependencies and have it launch a child container).
> --
> Kevin Sweeney
> @kts
>

--bcaec51b9dbb9aecfd051d4a4e39--