Mailing-List: contact issues-help@mesos.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@mesos.apache.org
Date: Wed, 15 Nov 2017 23:34:00 +0000 (UTC)
From: "Vinod Kone (JIRA)" <jira@apache.org>
To: issues@mesos.apache.org
Message-ID: <JIRA.13117148.1510189791000.250558.1510788840670@Atlassian.JIRA>
In-Reply-To: <JIRA.13117148.1510189791000@Atlassian.JIRA>
References: <JIRA.13117148.1510189791000@Atlassian.JIRA> <JIRA.13117148.1510189791435@jira-lw-us.apache.org>
Subject: [jira] [Commented] (MESOS-8185) Tasks can be known to the agent but
 unknown to the master.
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
archived-at: Wed, 15 Nov 2017 23:34:09 -0000


    [ https://issues.apache.org/jira/browse/MESOS-8185?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16254446#comment-16254446 ] 

Vinod Kone commented on MESOS-8185:
-----------------------------------

cc [~xujyan] [~megha.sharma] Is this still an issue after your changes?

> Tasks can be known to the agent but unknown to the master.
> ----------------------------------------------------------
>
>                 Key: MESOS-8185
>                 URL: https://issues.apache.org/jira/browse/MESOS-8185
>             Project: Mesos
>          Issue Type: Bug
>    Affects Versions: 1.2.0
>            Reporter: Ilya Pronin
>            Assignee: Ilya Pronin
>              Labels: reliability
>
> Currently, when a master re-registers an agent that was marked unreachable, it shutdowns all not partition-aware frameworks on that agent. When a master re-registers an agent that is already registered, it doesn't check that all tasks from the slave's re-registration message are known to it.
> It is possible that due to a transient loss of connectivity an agent may miss {{SlaveReregisteredMessage}} along with {{ShutdownFrameworkMessage}} and thus will not kill not partition-aware tasks. But the master will mark the agent as registered and will not re-add tasks that it thought will be killed. The agent may re-register again, this time successfully, before becoming marked unreachable while never having terminated tasks of not partition-aware frameworks. The master will simply forget those tasks ever existed, because it has "removed" them during the previous re-registration.
> Example scenario:
> # Connection from the master to the agent stops working
> # Agent doesn't see pings from the master and attempts to re-register
> # Master sends {{SlaveRegisteredMessage}} and {{ShutdownSlaveMessage}}, which don't get to the agent because of the connection failure. Agent is marked registered.
> # Network issue resolves, connection breaks. Agent retries re-registration.
> # Master thinks that the agent was registered since step (3) and just re-sends {{SlaveRegisteredMessage}}. Tasks remain running on the agent.
> One of the possible solutions would be to compare the list of tasks the the already registered agent reports in {{ReregisterSlaveMessage}} and the list of tasks the master has. In this case anything that the master doesn't know about should not exist on the agent.


--
This message was sent by Atlassian JIRA
(v6.4.14#64029)