mesos-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Qian Zhang (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (MESOS-5295) The task launched by non-checkpointed HTTP command executor will keep running till executor shutdown grace period (5s) after agent process exits.
Date Wed, 29 Jun 2016 05:42:45 GMT

    [ https://issues.apache.org/jira/browse/MESOS-5295?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15354585#comment-15354585
] 

Qian Zhang commented on MESOS-5295:
-----------------------------------

[~jvanz], sorry for the late. Yes, I am working on this, will post the patch soon :-)

> The task launched by non-checkpointed HTTP command executor will keep running till executor
shutdown grace period (5s) after agent process exits.
> -------------------------------------------------------------------------------------------------------------------------------------------------
>
>                 Key: MESOS-5295
>                 URL: https://issues.apache.org/jira/browse/MESOS-5295
>             Project: Mesos
>          Issue Type: Bug
>          Components: HTTP API
>            Reporter: Qian Zhang
>            Assignee: Qian Zhang
>
> When I test HTTP command executor, I found an issue, here is my steps:
> 1. A framework which has no checkpoint enabled launches a long running task (e.g., sleep
1000).
> 2. After the task is running, kill the agent.
>  
> Then I see the HTTP command executor will terminate after 5s ("DEFAULT_EXECUTOR_SHUTDOWN_GRACE_PERIOD"),
but the task will always run. This behavior is not consistent with driver based command executor:
after agent is killed, that executor will kill the task and then self terminate after 1s (there
is a "os::sleep(Seconds(1));" in "reaped()").
> The root cause of this difference is, for driver based command executor, when the driver
found agent is down, it will call executor->shutdown() (https://github.com/apache/mesos/blob/0.28.1/src/exec/exec.cpp#L487),
so the executor will kill the task and then self terminate. But for HTTP command executor,
its "disconnected()" will be called (https://github.com/apache/mesos/blob/0.28.1/src/executor/executor.cpp#L388)
when agent is down, and currently we do not do anything in its "disconnected()", so the task
will keep running and the executor will be killed after 5s (https://github.com/apache/mesos/blob/0.28.1/src/executor/executor.cpp#L623).
> The behavior of driver based command executor is correct, we need to make sure HTTP command
executor kills the task when agent is down if checkpoint is not enabled.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message