mesos-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Benjamin Mahler <benjamin.mah...@gmail.com>
Subject Re: Implementing an Executor
Date Mon, 01 Dec 2014 21:01:54 GMT
Sorry this is a bit of a tangent to the thread:

> For ex, if the executor crashes Mesos would get a TASK_LOST while the
container might still be running.

Mesos should destroy the container when the executor exits, are you seeing
otherwise?

> So we are doing something similar to Aurora's GC Executor to clean up
containers behind us.

Just to make sure you're aware, you don't need to do this with 0.21.0+, now
that we provide reconciliation support. :)

On Mon, Dec 1, 2014 at 12:56 PM, Diptanu Choudhury <diptanuc@gmail.com>
wrote:

> There are some downsides to this as well - For ex, if the executor crashes
> Mesos would get a TASK_LOST while the container might still be running. So
> we are doing something similar to Aurora's GC Executor to clean up
> containers behind us.
>
> On Mon, Dec 1, 2014 at 10:19 AM, Janet Borschowa <
> janet.borschowa@codefutures.com> wrote:
>
>> Hi Diptanu,
>> Thanks for your feedback. This sounds like the approach we'll be taking,
>> too.
>>
>>
>> On Sat, Nov 29, 2014 at 11:34 AM, Diptanu Choudhury <diptanuc@gmail.com>
>> wrote:
>>
>>> Hi Janet,
>>>
>>> We implemented the same in our Titan Mesos Executor. We have an executor
>>> manage multiple containers. We didn't want the overhead of running multiple
>>> executors to manage multiple containers. We are using RxJava heavily in our
>>> codebase, so whenever a launchTask callback is invoked in the MesosExecutor
>>> we publish an event and return immediately from the callback so that our
>>> executor can handle more callbacks from Mesos Slave.
>>>
>>> But the event which is published is picked up by the subscriber and that
>>> starts the workflow of downloading a container, mount volumes, sets up
>>> networking, sets up logging, and finally starts a monitoring process for
>>> the container. The monitoring process has the reference to the executor
>>> driver and we send back Mesos status updates whenever the state of the
>>> process changes.
>>>
>>>
>>> On Thu, Nov 20, 2014 at 8:40 AM, Janet Borschowa <
>>> janet.borschowa@codefutures.com> wrote:
>>>
>>>> Hi David,
>>>> Thanks for your suggestion. This approach sounds promising for what I
>>>> need to do. I'm going to have to try this out.
>>>>
>>>> Thanks!
>>>> Janet
>>>>
>>>> --
>>>>
>>>> Janet Borschowa
>>>> CodeFutures Corporation
>>>>
>>>> On Thu, Nov 20, 2014 at 5:04 AM, David Greenberg <
>>>> dsg123456789@gmail.com> wrote:
>>>>
>>>>> One cool feature of the docker containerizer is that you can actually
>>>>> launch your executor inside the docker image, so that you can just layer
>>>>> the executor's custom logic on top of whatever container you desire.
This
>>>>> way, you can more easily control what's happening in the docker image,
and
>>>>> still leverage the containerizer.
>>>>>
>>>>> On Thursday, November 20, 2014, Tim Chen <tim@mesosphere.io> wrote:
>>>>>
>>>>>> Hi Janet,
>>>>>>
>>>>>> Can you elaborate more what you like to get back from the docker
>>>>>> container that you launched?
>>>>>>
>>>>>> Thanks,
>>>>>>
>>>>>> Tim
>>>>>>
>>>>>> On Wed, Nov 19, 2014 at 5:22 PM, Tom Arnfeld <tom@duedil.com>
wrote:
>>>>>>
>>>>>>> Hi Janet,
>>>>>>>
>>>>>>> Oh sorry my mistake, I didn't read your email correctly, I thought
>>>>>>> you were using the containerizer. What you're doing here is actually
going
>>>>>>> to be quite difficult to do, the mesos docker containerizer has
some quite
>>>>>>> complex logic implemented to ensure the slave stays in sync with
the
>>>>>>> containers that are running, and kills anything that goes rogue.
>>>>>>>
>>>>>>> It's going to be non-trivial for you to do that from the executor,
>>>>>>> though I guess you could make use of the docker events API or
poll other
>>>>>>> endpoints in the API to check the status of your containers,
and off the
>>>>>>> back of that send status updates to the cluster. Doing this however
brings
>>>>>>> no guarantees that if your executor dies exceptionally (perhaps
OOMd) the
>>>>>>> containers spawned will die... they'll keep running in the background
and
>>>>>>> it'll be hard for you to know the state of your containers on
the cluster.
>>>>>>>
>>>>>>> You probably want to be aware (if you don't know already) that
the
>>>>>>> resource limits assigned to your tasks aren't going to be enforced
by mesos
>>>>>>> because docker is running outside of its control. You'll need
to pass the
>>>>>>> correct CPU/Memory limit parameters to your docker containers
to ensure
>>>>>>> this happens correctly.
>>>>>>>
>>>>>>> Here are the docker API docs;
>>>>>>> https://docs.docker.com/reference/api/docker_remote_api_v1.15/
>>>>>>>
>>>>>>> Something you might want to consider, if all you're trying to
do is
>>>>>>> allow your container access to details about itself (e.g `docker
inspect`)
>>>>>>> is to open up the docker remote API to be queried by your containers
on the
>>>>>>> slave, and switch to using the mesos docker containerizer.
>>>>>>>
>>>>>>> I hope that helps somewhat!
>>>>>>>
>>>>>>> Tom.
>>>>>>>
>>>>>>> --
>>>>>>>
>>>>>>> Tom Arnfeld
>>>>>>> Developer // DueDil
>>>>>>>
>>>>>>> (+44) 7525940046
>>>>>>> 25 Christopher Street, London, EC2A 2BS
>>>>>>>
>>>>>>>
>>>>>>> On Wed, Nov 19, 2014 at 10:16 PM, Janet Borschowa <
>>>>>>> janet.borschowa@codefutures.com> wrote:
>>>>>>>
>>>>>>>>  Hi,
>>>>>>>> I'm implementing an executor which is used by the mesos slave
to
>>>>>>>> launch tasks. The tasks are to launch a docker container
- this is because
>>>>>>>> I need more info about the launched container than what the
docker
>>>>>>>> containerizer returns.
>>>>>>>>
>>>>>>>> Is it OK to block in the executor's launchTask method until
the
>>>>>>>> task completes? If not, how does the framework discover when
that task
>>>>>>>> completes? I could spawn a process which notifies my executor
when the task
>>>>>>>> completes and then have my executor send a status update.
Or is there some
>>>>>>>> other recommended way to deal with this when the task could
run for an
>>>>>>>> indefinite period of time before completing its work?
>>>>>>>>
>>>>>>>> Thanks!
>>>>>>>>
>>>>>>>> Janet
>>>>>>>>
>>>>>>>> --
>>>>>>>>  Janet Borschowa
>>>>>>>> CodeFutures Corporation
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>
>>>>
>>>
>>>
>>> --
>>> Thanks,
>>> Diptanu Choudhury
>>> Web - www.linkedin.com/in/diptanu
>>> Twitter - @diptanu <http://twitter.com/diptanu>
>>>
>>
>>
>
>
> --
> Thanks,
> Diptanu Choudhury
> Web - www.linkedin.com/in/diptanu
> Twitter - @diptanu <http://twitter.com/diptanu>
>

Mime
View raw message