mesos-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Vinod Kone <vi...@twitter.com>
Subject Re: Architecture of Tasks vs. Executors
Date Fri, 12 Apr 2013 01:55:21 GMT
Yes, Mesos doesn't provide a fault-tolerant data store for frameworks out
of the box.

Having said that, Mesos does provide an abstraction called 'State' ( see
src/state/) that framework writers can use to provide reliability
guarantees.

You can see how AirBnB implemented their
Chronos<https://github.com/airbnb/chronos>framework (on top of Mesos)
to get an idea on how to use the State
abstraction.

Cheers,


@vinodkone


On Thu, Apr 11, 2013 at 6:37 PM, David Greenberg <dsg123456789@gmail.com>wrote:

> Hi Vinod! That clarifies things for me.
>
> Another thing that I don't understand is whether Mesos still provides a
> fault-tolerant data store for frameworks. I recall reading about it in the
> Mesos paper, but it seems like now I'd need to do my own thing with
> Zookeeper or some other concurrency-safe data store. Is that the case now?
>
> Thanks again,
> David
>
>
> On Tue, Apr 9, 2013 at 8:18 PM, Vinod Kone <vinod@twitter.com> wrote:
>
> > Hi David,
> >
> > The model of executors/tasks hasn't changed since we released 0.9.0.
> > That said, this is how things work.
> >
> > --> Framework registers with the master.
> > --> Master offers various slaves' resources to the framework.
> > --> Framework launches a task using the resources of an offer (an offer
> > maps uniquely to a slave).
> > --> When the slaves gets the task, it checks if the executor (see
> > TaskInfo) for that task is already running.
> > --> If yes, the task is sent to the executor and the executor driver
> calls
> > launchTask().
> > --> If no, the slave launches the executor and then sends the task to
> > the executor.
> > --> You can launch multiple executors per slave as long as they have
> > different ExecutorIDs.
> >
> > --> How the tasks and executors interact is up to the executor
> > (framework) writers.
> > --> From Mesos point of view a task is something that consumes some
> > resources.
> > --> If you are using cgroups isolation on the slave, Mesos ensures that
> > the cgroup container running executor+tasks is properly sized (cpu,
> > memory etc).
> >
> > --> Another thing to note is that sending terminal status update
> > (e.g., TASK_FINISHED) for a finished task is crucial, for Mesos to
> properly
> > reoffer those resources to the framework and resize the container (if
> > using cgroups).
> >
> > --> Finally, for framework writers that don't want to write a
> > specialized executor, Mesos has a default executor called Command
> > Executor. All it does is wrap your command line in a shell script and
> > run it.
> >
> > --> Note that, a command executor can only run 1 task. It exits right
> > after the
> > shell command returns.
> >
> > We are in the process of revamping the documentation. So hopefully
> > these things will be more clear soon.
> >
> > HTH,
> >
> >
> > @vinodkone
> >
> >
> > On Tue, Apr 9, 2013 at 3:05 PM, David Greenberg <dsg123456789@gmail.com>
> > wrote:
> > > I do not understand how tasks and executors work in the new mesos API.
> I
> > > assume that the model is still that a framework gets 1 scheduler, and
> > then
> > > that scheduler starts some number of executors. How do the executors
> and
> > > tasks interact, given that the scheduler only launches tasks? Are the
> > > executors implicitly created when needed, one per slave?
> > >
> > > Thanks!
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message