hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Jason Lowe (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (YARN-8648) Container cgroups are leaked when using docker
Date Tue, 14 Aug 2018 16:24:00 GMT

    [ https://issues.apache.org/jira/browse/YARN-8648?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16580025#comment-16580025
] 

Jason Lowe commented on YARN-8648:
----------------------------------

+1 for the proposal to fix the cgroup leak by having docker place its cgroup at the same level
YARN is creating its containers.  That preserves the semantic hierarchy of the hadoop-yarn
cgroup in case administrators are explicitly setting controls on the entire YARN container
hierarchy so it plays nice with other systems on the node.  It also preserves the cgroup sharing
semantics between nodes performing a mix of container launches where some are docker and others
are native.

Now the hard part is figuring out the cleanest way to have the NM create cgroups in the native
case and avoid creating the cgroup in the docker case.  This implies the container runtime
needs some say in how cgroups are handled if it isn't delegated to the container runtime completely.
 One potential issue with a useResourceHandlers() approach is if the NM wants to manipulate
cgroup settings on a live container.  Having a runtime that says it doesn't use resource handlers
implies that can't be done by that runtime, but it can be supported by the docker runtime.

We should consider breaking this up into two JIRAs if it proves difficult to hash through
the design.  It's a relatively small change to move the docker containers under the top-level
YARN cgroup hierarchy to fixes the cgroup leaks, with the side-effect that the NM continues
to create and cleanup unused cgroups per docker container launched.  We could follow up that
change with another JIRA to resolve the new design for the cgroup / container runtime interaction
so those empty cgroups are avoided in the docker case.  If we can hash it out quickly in one
JIRA that's great, but I want to make sure the leak problem doesn't linger while we work through
the architecture of cgroups and container runtimes.

> Container cgroups are leaked when using docker
> ----------------------------------------------
>
>                 Key: YARN-8648
>                 URL: https://issues.apache.org/jira/browse/YARN-8648
>             Project: Hadoop YARN
>          Issue Type: Bug
>            Reporter: Jim Brennan
>            Assignee: Jim Brennan
>            Priority: Major
>              Labels: Docker
>
> When you run with docker and enable cgroups for cpu, docker creates cgroups for all resources
on the system, not just for cpu.  For instance, if the {{yarn.nodemanager.linux-container-executor.cgroups.hierarchy=/hadoop-yarn}},
the nodemanager will create a cgroup for each container under {{/sys/fs/cgroup/cpu/hadoop-yarn}}.
 In the docker case, we pass this path via the {{--cgroup-parent}} command line argument.
  Docker then creates a cgroup for the docker container under that, for instance: {{/sys/fs/cgroup/cpu/hadoop-yarn/container_id/docker_container_id}}.
> When the container exits, docker cleans up the {{docker_container_id}} cgroup, and the
nodemanager cleans up the {{container_id}} cgroup,   All is good under {{/sys/fs/cgroup/hadoop-yarn}}.
> The problem is that docker also creates that same hierarchy under every resource under
{{/sys/fs/cgroup}}.  On the rhel7 system I am using, these are: blkio, cpuset, devices, freezer,
hugetlb, memory, net_cls, net_prio, perf_event, and systemd.    So for instance, docker creates
{{/sys/fs/cgroup/cpuset/hadoop-yarn/container_id/docker_container_id}}, but it only cleans
up the leaf cgroup {{docker_container_id}}.  Nobody cleans up the {{container_id}} cgroups
for these other resources.  On one of our busy clusters, we found > 100,000 of these leaked
cgroups.
> I found this in our 2.8-based version of hadoop, but I have been able to repro with current
hadoop.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

---------------------------------------------------------------------
To unsubscribe, e-mail: yarn-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: yarn-issues-help@hadoop.apache.org


Mime
View raw message