hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Abin Shahab (JIRA)" <j...@apache.org>
Subject [jira] [Created] (YARN-2466) Umbrella issue for Yarn launched Docker Containers
Date Fri, 29 Aug 2014 00:19:09 GMT
Abin Shahab created YARN-2466:

             Summary: Umbrella issue for Yarn launched Docker Containers
                 Key: YARN-2466
                 URL: https://issues.apache.org/jira/browse/YARN-2466
             Project: Hadoop YARN
          Issue Type: New Feature
    Affects Versions: 2.4.1
            Reporter: Abin Shahab
            Assignee: Abin Shahab

Docker (https://www.docker.io/) is, increasingly, a very popular container technology.

In context of YARN, the support for Docker will provide a very elegant solution to allow applications
to package their software into a Docker container (entire Linux file system incl. custom versions
of perl, python etc.) and use it as a blueprint to launch all their YARN containers with requisite
software environment. This provides both consistency (all YARN containers will have the same
software environment) and isolation (no interference with whatever is installed on the physical

In addition to software isolation mentioned above, Docker containers will provide resource,
network, and user-namespace isolation. 

Docker provides resource isolation through cgroups, similar to LinuxContainerExecutor. This
prevents one job from taking other jobs resource(memory and CPU) on the same hadoop cluster.

User-namespace isolation will ensure that the root on the container is mapped an unprivileged
user on the host. This is currently being added to Docker.

Network isolation will ensure that one user’s network traffic is completely isolated from
another user’s network traffic. 

Last but not the least, the interaction of Docker and Kerberos will have to be worked out.
These Docker containers must work in a secure hadoop environment.

Additional details are here: https://wiki.apache.org/hadoop/dineshs/IsolatingYarnAppsInDockerContainers

This message was sent by Atlassian JIRA

View raw message