mesos-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Alex Rukletsov <a...@mesosphere.io>
Subject Re: Trying to debug an issue in mesos task tracking
Date Fri, 23 Jan 2015 11:01:01 GMT
Itamar,

beyond checking master and slave logs, could you pleasse verify your
executor does send the TASK_FINISHED update? You may want to add some
logging and the check executor log. Mesos guarantees the delivery of
status updates, so I suspect the problem is on the executor's side.

On Wed, Jan 21, 2015 at 6:58 PM, Sharma Podila <spodila@netflix.com> wrote:
> Have you checked the mesos-slave and mesos-master logs for that task id?
> There should be logs in there for task state updates, including FINISHED.
> There can be specific cases where sometimes the task status is not reliably
> sent to your scheduler (due to mesos-master restarts, leader election
> changes, etc.). There is a task reconciliation support in Mesos. A periodic
> call to reconcile tasks from the scheduler can be helpful. There are also
> newer enhancements coming to the task reconciliation. In the mean time,
> there are other strategies such as what I use, which is periodic heartbeats
> from my custom executor to my scheduler (out of band). The timeouts for task
> runtimes are similar to heartbeats, except, you need a priori knowledge of
> all tasks' runtimes.
>
> Task runtime limits are not support inherently, as far as I know. Your
> executor can implement it, and that may be one simple way to do it. That
> could also be a good way to implement shell's rlimit*, in general.
>
>
>
> On Wed, Jan 21, 2015 at 1:22 AM, Itamar Ostricher <itamar@yowza3d.com>
> wrote:
>>
>> I'm using a custom internal framework, loosely based on MesosSubmit.
>> The phenomenon I'm seeing is something like this:
>> 1. Task X is assigned to slave S.
>> 2. I know this task should run for ~10minutes.
>> 3. On the master dashboard, I see that task X is in the "Running" state
>> for several *hours*.
>> 4. I SSH into slave S, and see that task X is *not* running. According to
>> the local logs on that slave, task X finished a long time ago, and seemed to
>> finish OK.
>> 5. According to the scheduler logs, it never got any update from task X
>> after the Staging->Running update.
>>
>> The phenomenon occurs pretty often, but it's not consistent or
>> deterministic.
>>
>> I'd appreciate your input on how to go about debugging it, and/or
>> implement a workaround to avoid wasted resources.
>>
>> I'm pretty sure the executor on the slave sends the TASK_FINISHED status
>> update (how can I verify that beyond my own logging?).
>> I'm pretty sure the scheduler never receives that update (again, how can I
>> verify that beyond my own logging?).
>> I have no idea if the master got the update and passed it through (how can
>> I check that?).
>> My scheduler and executor are written in Python.
>>
>> As for a workaround - setting a timeout on a task should do the trick. I
>> did not see any timeout field in the TaskInfo message. Does mesos support
>> the concept of per-task timeouts? Or should I implement my own task tracking
>> and timeouting mechanism in the scheduler?
>
>

Mime
View raw message