hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Ravi Prakash (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (YARN-2466) Umbrella issue for Yarn launched Docker Containers
Date Sat, 19 Nov 2016 00:49:59 GMT

     [ https://issues.apache.org/jira/browse/YARN-2466?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Ravi Prakash updated YARN-2466:
-------------------------------
    Description: 
Docker (https://www.docker.io/) is, increasingly, a very popular container technology.

In context of YARN, the support for Docker will provide a very elegant solution to allow applications
to package their software into a Docker container (entire Linux file system incl. custom versions
of perl, python etc.) and use it as a blueprint to launch all their YARN containers with requisite
software environment. This provides both consistency (all YARN containers will have the same
software environment) and isolation (no interference with whatever is installed on the physical
machine).

In addition to software isolation mentioned above, Docker containers will provide resource,
network, and user-namespace isolation. 

Docker provides resource isolation through cgroups, similar to LinuxContainerExecutor. This
prevents one job from taking other jobs resource(memory and CPU) on the same hadoop cluster.


User-namespace isolation will ensure that the root on the container is mapped an unprivileged
user on the host. This is currently being added to Docker.

Network isolation will ensure that one user’s network traffic is completely isolated from
another user’s network traffic. 

Last but not the least, the interaction of Docker and Kerberos will have to be worked out.
These Docker containers must work in a secure hadoop environment.

Additional details are here: https://wiki.apache.org/hadoop/dineshs/IsolatingYarnAppsInDockerContainers

  was:
*This has been deprecated and removed.* Please see https://issues.apache.org/jira/browse/YARN-5388
.

Docker (https://www.docker.io/) is, increasingly, a very popular container technology.

In context of YARN, the support for Docker will provide a very elegant solution to allow applications
to package their software into a Docker container (entire Linux file system incl. custom versions
of perl, python etc.) and use it as a blueprint to launch all their YARN containers with requisite
software environment. This provides both consistency (all YARN containers will have the same
software environment) and isolation (no interference with whatever is installed on the physical
machine).

In addition to software isolation mentioned above, Docker containers will provide resource,
network, and user-namespace isolation. 

Docker provides resource isolation through cgroups, similar to LinuxContainerExecutor. This
prevents one job from taking other jobs resource(memory and CPU) on the same hadoop cluster.


User-namespace isolation will ensure that the root on the container is mapped an unprivileged
user on the host. This is currently being added to Docker.

Network isolation will ensure that one user’s network traffic is completely isolated from
another user’s network traffic. 

Last but not the least, the interaction of Docker and Kerberos will have to be worked out.
These Docker containers must work in a secure hadoop environment.

Additional details are here: https://wiki.apache.org/hadoop/dineshs/IsolatingYarnAppsInDockerContainers


> Umbrella issue for Yarn launched Docker Containers
> --------------------------------------------------
>
>                 Key: YARN-2466
>                 URL: https://issues.apache.org/jira/browse/YARN-2466
>             Project: Hadoop YARN
>          Issue Type: New Feature
>    Affects Versions: 2.4.1
>            Reporter: Abin Shahab
>            Assignee: Abin Shahab
>
> Docker (https://www.docker.io/) is, increasingly, a very popular container technology.
> In context of YARN, the support for Docker will provide a very elegant solution to allow
applications to package their software into a Docker container (entire Linux file system incl.
custom versions of perl, python etc.) and use it as a blueprint to launch all their YARN containers
with requisite software environment. This provides both consistency (all YARN containers will
have the same software environment) and isolation (no interference with whatever is installed
on the physical machine).
> In addition to software isolation mentioned above, Docker containers will provide resource,
network, and user-namespace isolation. 
> Docker provides resource isolation through cgroups, similar to LinuxContainerExecutor.
This prevents one job from taking other jobs resource(memory and CPU) on the same hadoop cluster.

> User-namespace isolation will ensure that the root on the container is mapped an unprivileged
user on the host. This is currently being added to Docker.
> Network isolation will ensure that one user’s network traffic is completely isolated
from another user’s network traffic. 
> Last but not the least, the interaction of Docker and Kerberos will have to be worked
out. These Docker containers must work in a secure hadoop environment.
> Additional details are here: https://wiki.apache.org/hadoop/dineshs/IsolatingYarnAppsInDockerContainers



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: yarn-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: yarn-issues-help@hadoop.apache.org


Mime
View raw message