hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Craig Condit (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (YARN-8638) Allow linux container runtimes to be pluggable
Date Thu, 23 Aug 2018 16:39:00 GMT

    [ https://issues.apache.org/jira/browse/YARN-8638?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16590489#comment-16590489

Craig Condit commented on YARN-8638:

bq. I think this is the correct way to go and is what we should follow for the pluggable runtimes
as well. Using {{YARN_CONTAINER_RUNTIME_TYPE}} makes sense to me and I agree with your proposed

Perfect. I will prep a new patch and get it uploaded.

As an aside, the javadoc warnings are stumping me. According to everything I can find, referencing
constants in {{YarnConfiguration}} as we are in {\{\{@value\}}} tags should be legal (and
in fact seems to work in IDEA), but fails when using the maven javadoc plugin. Since we seem
to have several other occurrences in the same source files, I'm inclined to ignore the additional
warning for now.

> Allow linux container runtimes to be pluggable
> ----------------------------------------------
>                 Key: YARN-8638
>                 URL: https://issues.apache.org/jira/browse/YARN-8638
>             Project: Hadoop YARN
>          Issue Type: Improvement
>          Components: nodemanager
>    Affects Versions: 3.2.0
>            Reporter: Craig Condit
>            Assignee: Craig Condit
>            Priority: Minor
>         Attachments: YARN-8638.001.patch, YARN-8638.002.patch
> YARN currently supports three different Linux container runtimes (default, docker, and
javasandbox). However, it would be relatively straightforward to support arbitrary runtime
implementations. This would enable easier experimentation with new and emerging runtime technologies
(runc, containerd, etc.) without requiring a rebuild and redeployment of Hadoop. 
> This could be accomplished via a simple configuration change:
> {code:xml}
> <property>
>  <name>yarn.nodemanager.runtime.linux.allowed-runtimes</name>
>  <value>default,docker,experimental</value>
> </property>
> <property>
>  <name>yarn.nodemanager.runtime.linux.experimental.class</name>
>  <value>com.somecompany.yarn.runtime.ExperimentalLinuxContainerRuntime</value>
> </property>{code}
> In this example, {{yarn.nodemanager.runtime.linux.allowed-runtimes}} would now allow
arbitrary values. Additionally, {{yarn.nodemanager.runtime.linux.\{RUNTIME_KEY}.class}} would
indicate the {{LinuxContainerRuntime}} implementation to instantiate. A no-argument constructor
should be sufficient, as {{LinuxContainerRuntime}} already provides an {{initialize()}} method.
> {{DockerLinuxContainerRuntime.isDockerContainerRequested(Map<String, String> env)}}
and {{JavaSandboxLinuxContainerRuntime.isSandboxContainerRequested()}} could be generalized
to {{isRuntimeRequested(Map<String, String> env)}} and added to the {{LinuxContainerRuntime}} interface.
This would allow {{DelegatingLinuxContainerRuntime}} to select an appropriate runtime based
on whether that runtime claimed ownership of the current container execution.
> For backwards compatibility, the existing values (default,docker,javasandbox) would continue
to be supported as-is. Under the current logic, the evaluation order is javasandbox, docker,
default (with default being chosen if no other candidates are available). Under the new evaluation
logic, pluggable runtimes would be evaluated after docker and before default, in the order
in which they are defined in the allowed-runtimes list. This will change no behavior on current
clusters (as there would be no pluggable runtimes defined), and preserves behavior with respect
to ordering of existing runtimes.

This message was sent by Atlassian JIRA

To unsubscribe, e-mail: yarn-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: yarn-issues-help@hadoop.apache.org

View raw message