Mailing-List: contact issues-help@mesos.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@mesos.apache.org
Date: Wed, 16 Aug 2017 18:50:00 +0000 (UTC)
From: "Benjamin Mahler (JIRA)" <jira@apache.org>
To: issues@mesos.apache.org
Message-ID: <JIRA.13083651.1498788565000.46552.1502909400682@Atlassian.JIRA>
In-Reply-To: <JIRA.13083651.1498788565000@Atlassian.JIRA>
References: <JIRA.13083651.1498788565000@Atlassian.JIRA> <JIRA.13083651.1498788565608@jira-lw-us.apache.org>
Subject: [jira] [Commented] (MESOS-7744) Mesos Agent Sends TASK_KILL status
 update to Master, and still launches task
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: quoted-printable
archived-at: Wed, 16 Aug 2017 18:50:07 -0000


    [ https://issues.apache.org/jira/browse/MESOS-7744?page=3Dcom.atlassian=
.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=3D1612=
9261#comment-16129261 ]=20

Benjamin Mahler commented on MESOS-7744:
----------------------------------------

Ok, synced with [~alexr], looks like this will miss the 1.1.3 train and the=
re is no 1.1.4 planned.

> Mesos Agent Sends TASK_KILL status update to Master, and still launches t=
ask
> -------------------------------------------------------------------------=
---
>
>                 Key: MESOS-7744
>                 URL: https://issues.apache.org/jira/browse/MESOS-7744
>             Project: Mesos
>          Issue Type: Bug
>    Affects Versions: 1.0.1
>            Reporter: Sargun Dhillon
>            Assignee: Benjamin Mahler
>            Priority: Critical
>              Labels: reliability
>
> We sometimes launch jobs, and cancel them in ~7 seconds, if we don't get =
a TASK_STARTING back from the agent. Under certain conditions it can result=
 in Mesos losing track of the task. The chunk of the logs which is interest=
ing is here:
> {code}
> Jun 29 23:22:26 titusagent-mainvpc-r3.8xlarge.2-i-04907efc9f1f8535c mesos=
-slave[4290]: I0629 23:22:26.951799  5171 slave.cpp:1495] Got assigned task=
 Titus-7590548-worker-0-4476 for framework TitusFramework
> Jun 29 23:22:26 titusagent-mainvpc-r3.8xlarge.2-i-04907efc9f1f8535c mesos=
-slave[4290]: I0629 23:22:26.952251  5171 slave.cpp:1614] Launching task Ti=
tus-7590548-worker-0-4476 for framework TitusFramework
> Jun 29 23:22:37 titusagent-mainvpc-r3.8xlarge.2-i-04907efc9f1f8535c mesos=
-slave[4290]: I0629 23:22:37.484611  5171 slave.cpp:1853] Queuing task =E2=
=80=98Titus-7590548-worker-0-4476=E2=80=99 for executor =E2=80=98docker-exe=
cutor=E2=80=99 of framework TitusFramework at executor(1)@100.66.11.10:1770=
7
> Jun 29 23:22:37 titusagent-mainvpc-r3.8xlarge.2-i-04907efc9f1f8535c mesos=
-slave[4290]: I0629 23:22:37.487876  5171 slave.cpp:2035] Asked to kill tas=
k Titus-7590548-worker-0-4476 of framework TitusFramework
> Jun 29 23:22:37 titusagent-mainvpc-r3.8xlarge.2-i-04907efc9f1f8535c mesos=
-slave[4290]: I0629 23:22:37.488994  5171 slave.cpp:3211] Handling status u=
pdate TASK_KILLED (UUID: 898215d6-a244-4dbe-bc9c-878a22d36ea4) for task Tit=
us-7590548-worker-0-4476 of framework TitusFramework from @0.0.0.0:0
> Jun 29 23:22:37 titusagent-mainvpc-r3.8xlarge.2-i-04907efc9f1f8535c mesos=
-slave[4290]: I0629 23:22:37.490603  5171 slave.cpp:2005] Sending queued ta=
sk =E2=80=98Titus-7590548-worker-0-4476=E2=80=99 to executor =E2=80=98docke=
r-executor=E2=80=99 of framework TitusFramework at executor(1)@100.66.11.10=
:17707{
> {code}
> In our executor, we see that the launch message arrives after the master =
has already gotten the kill update. We then send non-terminal state updates=
 to the agent, and yet it doesn't forward these to our framework. We're usi=
ng a custom executor which is based on the older mesos-go bindings.=20


--
This message was sent by Atlassian JIRA
(v6.4.14#64029)