mesos-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Plotka, Bartlomiej" <bartlomiej.plo...@intel.com>
Subject RE: Custom python executor with Docker
Date Tue, 10 Nov 2015 15:15:56 GMT
Hi All,

I can see there are many use cases for extending Docker tasks with some pre/post work.

In our solution we want to prepare and attach some volumes (e.g ceph volumes) to Docker. As
you know we cannot (easily) attach a volume to the running Docker, so we need pre script for
preparing & attaching in one task. Unfortunately, we also need to clean-up these volumes
and it seems there is not any “slavePostLaunchDockerHook” available now in Mesos.

We’ve developed short script for running such tasks using common Mesos Containerizer and
without any hook module in Mesos. We can just put it in the Marathon “command” parameter.
Do you think such created Docker container will have limited cpu shares & memory (using
Mesos resources)?

Here is pseudo-bash code:
#!/bin/bash

function cleanup {{
  docker rm <docker_name>;
 <clean up volumes>
}}

trap cleanup SIGKILL SIGSTOP SIGHUP SIGINT SIGTERM;

<prepare volumes>
docker run --name <docker-name> -p <port>:22 <volumes> <docker image>
bash -c "<docker cmd>";

We are aware that there wouldn’t be any sandbox support and any other Docker Containerizer
features. However, our main concern is about cgroups… We are not sure if our newly created
Docker (via this script) will inherit our task’s cgroups and as a result will limit memory,
cpus via default Mesos Task’s resource parameters.

Anyone knows the answer?

As far as I understand without  special configuration ‘Docker run’ somehow pass the ‘run’
command to the docker daemon so we don’t have control of container cgroup…

Kind Regards,
Bartek Plotka (bplotka)

From: Kevin Sweeney [mailto:kevints@apache.org]
Sent: Wednesday, August 12, 2015 1:52 AM
To: user@mesos.apache.org
Subject: Re: Custom python executor with Docker

Apache Aurora [1] uses a custom Python executor and supports Docker via the containerizer.
There's just one problem - the container has to have a Python2.7 runtime inside that can run
the executor PEX file [2]. So if you're okay with that restriction you're in business (and
you can use the Aurora configuration DSL to describe setup/teardown steps).

[1] https://aurora.apache.org
[2] https://pex.readthedocs.org/en/latest/

On Tue, Aug 11, 2015 at 4:42 PM, Tim Chen <tim@mesosphere.io<mailto:tim@mesosphere.io>>
wrote:
So currently there is a review out for pre-hooks (https://reviews.apache.org/r/36185/) before
a docker container launches.

We can also add a post hook, but like to see if the specified hook satifies what you guys
are looking for.

Tim

On Tue, Aug 11, 2015 at 4:28 PM, Tom Fordon <tom.fordon@gmail.com<mailto:tom.fordon@gmail.com>>
wrote:
We ended up implementing a solution where we did the pre/post steps as separate mesos tasks
and adding logic to our scheduler to ensure they were run on the same machine.  If anybody
knows of a standard / openly available DockerExecutor like what is described below, my team
would be greatly interested.


On Fri, Aug 7, 2015 at 4:01 AM, Kapil Malik <kmalik@adobe.com<mailto:kmalik@adobe.com>>
wrote:
Hi,

We have a similar usecase while running multi-user workloads on mesos. Users provide docker
images encapsulating application logic, which we (we = say some “Central API”) schedule
on Chronos / Marathon. However, we need to run some standard pre / post steps for every docker
submitted by users. We have following options –


1.       Ask every user to embed their logic inside a pre-defined docker template which will
perform pre/post steps.

==> This is error prone, makes us dependent on whether the users followed template, and
not very popular with users either.



2.       Extend every user docker (FROM <>) and find a way to add pre-post steps in
our docker. Refer this docker when scheduling on chronos / marathon.

==> Building new dockers does not scale as users and applications grow



3.       Write a custom executor which will perform the pre-post steps and manage the user
docker lifetime.

==> Deals with user docker lifetime and is obviously complex.

Is there a standard / openly available DockerExecutor which manages the docker lifetime and
which I can extend to build my custom executor? This way I will be concerned only with my
custom logic (pre/post steps) and still get benefits of a standard way to manage docker containers.

Btw, thanks for the meaningful discussion below, it is very helpful.

Thanks and regards,

Kapil Malik | kmalik@adobe.com<mailto:kmalik@adobe.com> | 33430 / 8800836581

From: James DeFelice [mailto:james.defelice@gmail.com<mailto:james.defelice@gmail.com>]
Sent: 09 April 2015 18:12
To: user@mesos.apache.org<mailto:user@mesos.apache.org>
Subject: Re: Custom python executor with Docker

If you can run the pre/post steps in a container then I'd recommend building a Docker image
that includes your pre/post step scripting + your algorithm and launching it using the built-in
mesos Docker containerizer. It's much simpler than managing the lifetime of the Docker container
yourself.

On Thu, Apr 9, 2015 at 8:29 AM, Tom Fordon <tom.fordon@gmail.com<mailto:tom.fordon@gmail.com>>
wrote:
Thanks for all the responses, I really appreciate the help.  Let me try to state my problem
more clearly

Our project is performing file-based data processing.  I would like to keep the actual algorithm
as contained as possible since we are in an R&D setting and will be getting untested code.
 We have some pre/post steps that need to be run on the same box as the actual algorithm:
downloading/uploading files and database calls.

We can run the pre/post steps and algorithm within the same container.  The algorithm will
be a little less contained, but it will work.

Docker letting you specify a cgroup parent is really exciting.  If I invoke a docker container
with the executor as the cgroup-parent are there any other steps I need to perform?  Would
I need to do anything special to make mesos aware of the resource usage, or is that handled
since the docker process would be in the executors cgroup?

Thanks again,
Tom

On Tue, Apr 7, 2015 at 8:10 PM, Timothy Chen <tnachen@gmail.com<mailto:tnachen@gmail.com>>
wrote:
Hi Tom(s),

Tom Arnfeld is right, if you want to launch your own docker container
in your custom executor you will have to handle all the issues
yourself and not able to use the Docker containerizer at all.

Alternatively, you can actually launch your custom executor in a
Docker container by Mesos, by specifying the ContainerInfo in the
ExecutorInfo.
What this means is that your custom executor is already running in a
docker container, and you can do your custom logic afterwards. This
does means you can simply just launch multiple containers in the
executor anymore.

If there is something you want to do and doesnt' fit these let us know
what you're trying to achieve and we can see what we can do.

Tim

On Tue, Apr 7, 2015 at 4:15 PM, Tom Arnfeld <tom@duedil.com<mailto:tom@duedil.com>>
wrote:
> It's not possible to invoke the docker containerizer from outside of Mesos,
> as far as I know.
>
> If you persue this route, you can run into issues with orphaned containers
> as your executor may die for some unknown reason, and the container is still
> running. Recovering from this can be tricky business, so it's better if you
> can adapt your framework design to fit within the Mesos Task/Executor
> pattern.
>
> --
>
> Tom Arnfeld
> Developer // DueDil
>
> (+44) 7525940046<tel:%28%2B44%29%207525940046>
> 25 Christopher Street, London, EC2A 2BS
>
>
> On Mon, Apr 6, 2015 at 7:00 PM, Vinod Kone <vinodkone@apache.org<mailto:vinodkone@apache.org>>
wrote:
>>
>> Tim, do you want answer this?
>>
>> On Wed, Apr 1, 2015 at 7:27 AM, Tom Fordon <tom.fordon@gmail.com<mailto:tom.fordon@gmail.com>>
wrote:
>>>
>>> Hi.  I'm trying to understand using docker within a custom executor. For
>>> each of my tasks, I would like to perform some steps on the node before
>>> launching a docker container. I was planning on writing a custom python
>>> executor for this, but I wasn't sure how to launch docker from within this
>>> executor.
>>>
>>> Can I just call docker in a subprocess using the ContainerInfo from the
>>> Task? If I do this, how does the Containerizer fit in?
>>>
>>> Thank you,
>>> Tom Fordon
>>
>>
>




--
James DeFelice
585.241.9488<tel:585.241.9488> (voice)
650.649.6071<tel:650.649.6071> (fax)



--------------------------------------------------------------------

Intel Technology Poland sp. z o.o.
ul. Slowackiego 173 | 80-298 Gdansk | Sad Rejonowy Gdansk Polnoc | VII Wydzial Gospodarczy
Krajowego Rejestru Sadowego - KRS 101882 | NIP 957-07-52-316 | Kapital zakladowy 200.000 PLN.

Ta wiadomosc wraz z zalacznikami jest przeznaczona dla okreslonego adresata i moze zawierac
informacje poufne. W razie przypadkowego otrzymania tej wiadomosci, prosimy o powiadomienie
nadawcy oraz trwale jej usuniecie; jakiekolwiek
przegladanie lub rozpowszechnianie jest zabronione.
This e-mail and any attachments may contain confidential material for the sole use of the
intended recipient(s). If you are not the intended recipient, please contact the sender and
delete all copies; any review or distribution by
others is strictly prohibited.
Mime
View raw message