mesos-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Benjamin Mahler <benjamin.mah...@gmail.com>
Subject Re: Implementing an Executor
Date Mon, 01 Dec 2014 21:47:40 GMT
Got it, thanks for clarifying. :)

On Mon, Dec 1, 2014 at 1:40 PM, Diptanu Choudhury <diptanuc@gmail.com>
wrote:

> Hey Ben,
>
> We are not using the Docker containerizer from Mesos Core yet, we just
> have a plain executor. And by "container" I meant the docker container
> here. The docker containers in this case are started by the executor [which
> uses Docker's remote API]. Mesos does kill the container that it creates to
> run the executor but however the docker containers are not controlled by
> Mesos Slave.
>
> Also because Mesos doesn't know the existence of those containers because
> the executor was monitoring them, task reconciliation won't help either.
> This is another downside of this approach. So we use an out-of-band
> communication here to kill containers that the Framework and Mesos Master
> thinks should not be running.
>
> Our main problem is - we want the docker containers to be setup in a
> certain way, for example attach elastic network interfaces to containers,
> mount ZFS volumes etc, and that's why we had to write a custom executor in
> the first place.
>
>
> On Mon, Dec 1, 2014 at 1:01 PM, Benjamin Mahler <benjamin.mahler@gmail.com
> > wrote:
>
>> Sorry this is a bit of a tangent to the thread:
>>
>> > For ex, if the executor crashes Mesos would get a TASK_LOST while the
>> container might still be running.
>>
>> Mesos should destroy the container when the executor exits, are you
>> seeing otherwise?
>>
>> > So we are doing something similar to Aurora's GC Executor to clean up
>> containers behind us.
>>
>> Just to make sure you're aware, you don't need to do this with 0.21.0+,
>> now that we provide reconciliation support. :)
>>
>> On Mon, Dec 1, 2014 at 12:56 PM, Diptanu Choudhury <diptanuc@gmail.com>
>> wrote:
>>
>>> There are some downsides to this as well - For ex, if the executor
>>> crashes Mesos would get a TASK_LOST while the container might still be
>>> running. So we are doing something similar to Aurora's GC Executor to clean
>>> up containers behind us.
>>>
>>> On Mon, Dec 1, 2014 at 10:19 AM, Janet Borschowa <
>>> janet.borschowa@codefutures.com> wrote:
>>>
>>>> Hi Diptanu,
>>>> Thanks for your feedback. This sounds like the approach we'll be
>>>> taking, too.
>>>>
>>>>
>>>> On Sat, Nov 29, 2014 at 11:34 AM, Diptanu Choudhury <diptanuc@gmail.com
>>>> > wrote:
>>>>
>>>>> Hi Janet,
>>>>>
>>>>> We implemented the same in our Titan Mesos Executor. We have an
>>>>> executor manage multiple containers. We didn't want the overhead of running
>>>>> multiple executors to manage multiple containers. We are using RxJava
>>>>> heavily in our codebase, so whenever a launchTask callback is invoked
in
>>>>> the MesosExecutor we publish an event and return immediately from the
>>>>> callback so that our executor can handle more callbacks from Mesos Slave.
>>>>>
>>>>> But the event which is published is picked up by the subscriber and
>>>>> that starts the workflow of downloading a container, mount volumes, sets
up
>>>>> networking, sets up logging, and finally starts a monitoring process
for
>>>>> the container. The monitoring process has the reference to the executor
>>>>> driver and we send back Mesos status updates whenever the state of the
>>>>> process changes.
>>>>>
>>>>>
>>>>> On Thu, Nov 20, 2014 at 8:40 AM, Janet Borschowa <
>>>>> janet.borschowa@codefutures.com> wrote:
>>>>>
>>>>>> Hi David,
>>>>>> Thanks for your suggestion. This approach sounds promising for what
I
>>>>>> need to do. I'm going to have to try this out.
>>>>>>
>>>>>> Thanks!
>>>>>> Janet
>>>>>>
>>>>>> --
>>>>>>
>>>>>> Janet Borschowa
>>>>>> CodeFutures Corporation
>>>>>>
>>>>>> On Thu, Nov 20, 2014 at 5:04 AM, David Greenberg <
>>>>>> dsg123456789@gmail.com> wrote:
>>>>>>
>>>>>>> One cool feature of the docker containerizer is that you can
>>>>>>> actually launch your executor inside the docker image, so that
you can just
>>>>>>> layer the executor's custom logic on top of whatever container
you desire.
>>>>>>> This way, you can more easily control what's happening in the
docker image,
>>>>>>> and still leverage the containerizer.
>>>>>>>
>>>>>>> On Thursday, November 20, 2014, Tim Chen <tim@mesosphere.io>
wrote:
>>>>>>>
>>>>>>>> Hi Janet,
>>>>>>>>
>>>>>>>> Can you elaborate more what you like to get back from the
docker
>>>>>>>> container that you launched?
>>>>>>>>
>>>>>>>> Thanks,
>>>>>>>>
>>>>>>>> Tim
>>>>>>>>
>>>>>>>> On Wed, Nov 19, 2014 at 5:22 PM, Tom Arnfeld <tom@duedil.com>
>>>>>>>> wrote:
>>>>>>>>
>>>>>>>>> Hi Janet,
>>>>>>>>>
>>>>>>>>> Oh sorry my mistake, I didn't read your email correctly,
I thought
>>>>>>>>> you were using the containerizer. What you're doing here
is actually going
>>>>>>>>> to be quite difficult to do, the mesos docker containerizer
has some quite
>>>>>>>>> complex logic implemented to ensure the slave stays in
sync with the
>>>>>>>>> containers that are running, and kills anything that
goes rogue.
>>>>>>>>>
>>>>>>>>> It's going to be non-trivial for you to do that from
the executor,
>>>>>>>>> though I guess you could make use of the docker events
API or poll other
>>>>>>>>> endpoints in the API to check the status of your containers,
and off the
>>>>>>>>> back of that send status updates to the cluster. Doing
this however brings
>>>>>>>>> no guarantees that if your executor dies exceptionally
(perhaps OOMd) the
>>>>>>>>> containers spawned will die... they'll keep running in
the background and
>>>>>>>>> it'll be hard for you to know the state of your containers
on the cluster.
>>>>>>>>>
>>>>>>>>> You probably want to be aware (if you don't know already)
that the
>>>>>>>>> resource limits assigned to your tasks aren't going to
be enforced by mesos
>>>>>>>>> because docker is running outside of its control. You'll
need to pass the
>>>>>>>>> correct CPU/Memory limit parameters to your docker containers
to ensure
>>>>>>>>> this happens correctly.
>>>>>>>>>
>>>>>>>>> Here are the docker API docs;
>>>>>>>>> https://docs.docker.com/reference/api/docker_remote_api_v1.15/
>>>>>>>>>
>>>>>>>>> Something you might want to consider, if all you're trying
to do
>>>>>>>>> is allow your container access to details about itself
(e.g `docker
>>>>>>>>> inspect`) is to open up the docker remote API to be queried
by your
>>>>>>>>> containers on the slave, and switch to using the mesos
docker containerizer.
>>>>>>>>>
>>>>>>>>> I hope that helps somewhat!
>>>>>>>>>
>>>>>>>>> Tom.
>>>>>>>>>
>>>>>>>>> --
>>>>>>>>>
>>>>>>>>> Tom Arnfeld
>>>>>>>>> Developer // DueDil
>>>>>>>>>
>>>>>>>>> (+44) 7525940046
>>>>>>>>> 25 Christopher Street, London, EC2A 2BS
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> On Wed, Nov 19, 2014 at 10:16 PM, Janet Borschowa <
>>>>>>>>> janet.borschowa@codefutures.com> wrote:
>>>>>>>>>
>>>>>>>>>>  Hi,
>>>>>>>>>> I'm implementing an executor which is used by the
mesos slave to
>>>>>>>>>> launch tasks. The tasks are to launch a docker container
- this is because
>>>>>>>>>> I need more info about the launched container than
what the docker
>>>>>>>>>> containerizer returns.
>>>>>>>>>>
>>>>>>>>>> Is it OK to block in the executor's launchTask method
until the
>>>>>>>>>> task completes? If not, how does the framework discover
when that task
>>>>>>>>>> completes? I could spawn a process which notifies
my executor when the task
>>>>>>>>>> completes and then have my executor send a status
update. Or is there some
>>>>>>>>>> other recommended way to deal with this when the
task could run for an
>>>>>>>>>> indefinite period of time before completing its work?
>>>>>>>>>>
>>>>>>>>>> Thanks!
>>>>>>>>>>
>>>>>>>>>> Janet
>>>>>>>>>>
>>>>>>>>>> --
>>>>>>>>>>  Janet Borschowa
>>>>>>>>>> CodeFutures Corporation
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> Thanks,
>>>>> Diptanu Choudhury
>>>>> Web - www.linkedin.com/in/diptanu
>>>>> Twitter - @diptanu <http://twitter.com/diptanu>
>>>>>
>>>>
>>>>
>>>
>>>
>>> --
>>> Thanks,
>>> Diptanu Choudhury
>>> Web - www.linkedin.com/in/diptanu
>>> Twitter - @diptanu <http://twitter.com/diptanu>
>>>
>>
>>
>
>
> --
> Thanks,
> Diptanu Choudhury
> Web - www.linkedin.com/in/diptanu
> Twitter - @diptanu <http://twitter.com/diptanu>
>

Mime
View raw message