hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Eric Yang (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (YARN-9074) Docker container rm command should be executed after stop
Date Fri, 11 Jan 2019 17:30:00 GMT

    [ https://issues.apache.org/jira/browse/YARN-9074?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16740577#comment-16740577
] 

Eric Yang commented on YARN-9074:
---------------------------------

[~uranus] During the design phase, the defined transition has one small difference as listed
in this table:

| Transition | Remove workdir | Remove Docker Image |
| Reinitialize | No | Yes |
| Task Finished | Yes | Yes |
| Task Killed | Yes | Yes |

Workdir should not be removed, if the container is reinitialized.  This is best effort to
keep task intermediate data.  However, container-executor code does not work with directory
that is already existed.  Therefore, node manager also removes workdir for reinitialization
in practice.  Hence, your suggested optimization can work for all cases.  I am inclined to
include this patch, if [~shanekumpf@gmail.com] agrees.

> Docker container rm command should be executed after stop
> ---------------------------------------------------------
>
>                 Key: YARN-9074
>                 URL: https://issues.apache.org/jira/browse/YARN-9074
>             Project: Hadoop YARN
>          Issue Type: Sub-task
>            Reporter: Zhaohui Xin
>            Assignee: Zhaohui Xin
>            Priority: Major
>         Attachments: YARN-9074.001.patch, image-2018-12-01-11-36-12-448.png, image-2018-12-01-11-38-18-191.png
>
>
> {code:java}
> @Override
> public void transition(ContainerImpl container, ContainerEvent event) {
> container.setIsReInitializing(false);
> // Set exit code to 0 on success 
> container.exitCode = 0;
> // TODO: Add containerWorkDir to the deletion service.
> if (DockerLinuxContainerRuntime.isDockerContainerRequested(
> container.daemonConf,
> container.getLaunchContext().getEnvironment())) {
> removeDockerContainer(container);
> }
> if (clCleanupRequired) {
> container.dispatcher.getEventHandler().handle(
> new ContainersLauncherEvent(container,
> ContainersLauncherEventType.CLEANUP_CONTAINER));
> }
> container.cleanup();
> }{code}
> Now, when container is finished, NM firstly execute "_docker rm xxx"_  to remove it
and this thread is placed in DeletionService. see more in YARN-5366 .
> Next, NM will execute "_docker stop_" and "docker kill" command. these tow commands are
wrapped up in ContainerCleanup thread and executed by ContainersLauncher. see more in YARN-7644. 
> The above will cause the container's cleanup to be split into two threads. I think we
should refactor these code to make all docker container killing process be place in ContainerCleanup
thread and "_docker rm_" should be executed last.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

---------------------------------------------------------------------
To unsubscribe, e-mail: yarn-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: yarn-issues-help@hadoop.apache.org


Mime
View raw message