hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Allen Wittenauer (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (YARN-1964) Create Docker analog of the LinuxContainerExecutor in YARN
Date Thu, 19 Jun 2014 17:16:26 GMT

    [ https://issues.apache.org/jira/browse/YARN-1964?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14037550#comment-14037550

Allen Wittenauer commented on YARN-1964:

I took a look at the patch.  After some discussion/clarification from Abin and Dinesh, here
are some issues we've identified with the current patch:

* yarn-site.xml names are not in the yarn.* hierarchy.

* The environment variables being imported are likely incomplete (HADOOP_PREFIX, at a minimum,
should be added to the list).  

* Application Masters cannot currently be run from DockerContainerExecutor.  The current patch
launches them with the default launcher.  

* There isn't any special distributed cache handling, which could/will expose private data
to everyone as well as risk confusing the node manager if the distributed cache contents are
changed out from underneath it.  

* Same thing with logging.

* This hasn't been tested with any level of security (either 'run as the user' or fully kerberized).
 There are likely more holes/problems that will be discovered after those tests.

* There is a question as to how to handle importing Java and the Hadoop setup into the container.
There is a risk that just adding HADOOP_PREFIX and marking it as RO may break things in certain
configurations (specifically logs, tmp, and pids dirs).

> Create Docker analog of the LinuxContainerExecutor in YARN
> ----------------------------------------------------------
>                 Key: YARN-1964
>                 URL: https://issues.apache.org/jira/browse/YARN-1964
>             Project: Hadoop YARN
>          Issue Type: New Feature
>    Affects Versions: 2.2.0
>            Reporter: Arun C Murthy
>            Assignee: Abin Shahab
>         Attachments: yarn-1964-branch-2.2.0-docker.patch, yarn-1964-branch-2.2.0-docker.patch,
yarn-1964-docker.patch, yarn-1964-docker.patch, yarn-1964-docker.patch, yarn-1964-docker.patch,
> Docker (https://www.docker.io/) is, increasingly, a very popular container technology.
> In context of YARN, the support for Docker will provide a very elegant solution to allow
applications to *package* their software into a Docker container (entire Linux file system
incl. custom versions of perl, python etc.) and use it as a blueprint to launch all their
YARN containers with requisite software environment. This provides both consistency (all YARN
containers will have the same software environment) and isolation (no interference with whatever
is installed on the physical machine).

This message was sent by Atlassian JIRA

View raw message