hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Jason Lowe (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (YARN-6930) Admins should be able to explicitly enable specific LinuxContainerRuntime in the NodeManager
Date Thu, 17 Aug 2017 16:39:00 GMT

    [ https://issues.apache.org/jira/browse/YARN-6930?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16130786#comment-16130786

Jason Lowe commented on YARN-6930:

Sorry to jump in a bit late here.  I agree that going the reflection route is a bit dubious
at this point for a couple of reasons.  This isn't a supported plugin interface yet, meaning
we don't expect users to provide their own runtime classes which would require reflection
to load.  Also the runtime interface would need at least a method to determine whether that
runtime wants to support a particular container.  Currently DelegatingLinuxContainerRuntime#pickContainerRuntime
has very specific knowledge of all the various runtimes and is doing that logic.  It would
need to be changed to an ordered list where the first runtime to say it wants to support a
container gets it.  So I think we're OK keeping it hardcoded for now given it's not really
a general interface yet.  Even if we want to migrate to a full plugin interface later that
requires fully-qualified class names in the property list, we can always continue to recognize
the legacy values for backwards compatibility.

On a related note, how does a user know what values are valid for yarn.nodemanager.runtime.linux.allowed-runtimes?
 If we are going the hardcoded route then arguably we should list the ones we know about here
so users know what they can do.  Also from a performance standpoint, if we're going hardcoded
then it would be more efficient to use enums for the various supported runtimes and use an
EnumSet to track allowed runtimes.  Then we aren't doing a linear search with string compares
for each runtime on each container operation but rather a simple bitmask check.  Not a must-fix
thing, just pointing out the possibility.

We could also explicitly decouple the config values from the source, using simpler values
like "default,docker,javasandbox" rather than the class name.  The latter has the drawback
that if anyone decides to refactor the class name they just inadvertently broke all existing
user configs that referred to it.  If we keep an explicit mapping of values to classnames
(like the enum) then this can't happen.

> Admins should be able to explicitly enable specific LinuxContainerRuntime in the NodeManager
> --------------------------------------------------------------------------------------------
>                 Key: YARN-6930
>                 URL: https://issues.apache.org/jira/browse/YARN-6930
>             Project: Hadoop YARN
>          Issue Type: Improvement
>          Components: nodemanager
>            Reporter: Vinod Kumar Vavilapalli
>            Assignee: Shane Kumpf
>         Attachments: YARN-6930.001.patch, YARN-6930.002.patch, YARN-6930.003.patch
> Today, in the java land, all LinuxContainerRuntimes are always enabled when using LinuxContainerExecutor
and the user can simply invoke anything that he/she wants - default, docker, java-sandbox.
> We should have a way for admins to explicitly enable only specific runtimes that he/she
decides for the cluster. And by default, we should have everything other than the default
one disabled.

This message was sent by Atlassian JIRA

To unsubscribe, e-mail: yarn-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: yarn-issues-help@hadoop.apache.org

View raw message