hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Nathan Roberts (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (YARN-2904) Use linux cgroups to enhance container tear down
Date Fri, 16 Dec 2016 17:09:58 GMT

    [ https://issues.apache.org/jira/browse/YARN-2904?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15754975#comment-15754975
] 

Nathan Roberts commented on YARN-2904:
--------------------------------------

Simple streaming job that does the following illustrates tasks escaping. (/usr/bin/timeout
does a setpgrp() which puts it in its own session). 
{noformat}
#!/bin/bash
/usr/bin/timeout 1d /bin/sleep 1000
{noformat}

Mesos has apparently addressed this a couple of different ways including 1) freeze_container->kill_all_processes_in_container->unfreeze_container;
or 2) use a private PID NS within the container and then kill PID1 within the container. 

> Use linux cgroups to enhance container tear down
> ------------------------------------------------
>
>                 Key: YARN-2904
>                 URL: https://issues.apache.org/jira/browse/YARN-2904
>             Project: Hadoop YARN
>          Issue Type: Improvement
>          Components: nodemanager
>    Affects Versions: 2.6.0
>            Reporter: Nathan Roberts
>
> If we are launching yarn containers within cgroups, linux provides some guarantees that
can help completely tear down a container.  Specifically, linux guarantees that tasks can't
escape a cgroup. We can use this fact to tear down a yarn container without leaking tasks.
> Today, a SIGTERM is sent to the session (normally lead by bash). When the session leader
exits, the LCE sees this and assumes all resources have been given back to the system. This
is not guaranteed. Example: YARN-2809 implements a workaround that is only necessary because
tasks are still lingering within the cgroup when the nodemanager attempts to delete it.  



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: yarn-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: yarn-issues-help@hadoop.apache.org


Mime
View raw message