hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Sangjin Lee (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (YARN-4577) Enable aux services to have their own custom classpath/jar file
Date Fri, 15 Jan 2016 00:30:40 GMT

    [ https://issues.apache.org/jira/browse/YARN-4577?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15099191#comment-15099191

Sangjin Lee commented on YARN-4577:

The use case here is simple: if we specify the aux-services classpath, either from local fs
or from hdfs, we will load this service from the specified classpath (no matter we set the
classpath in NM path or not). Otherwise, we load the service from the NM path.

Hmm, is one of the goals to preserve aux service's dependencies against hadoop's dependencies
(as I see in the linked ticket SPARK-12807)? If so, I don't think the current approach in
the patch does that. Note that URLClassLoader (or any simple extension of ClassLoader) always
*delegates classloading to the parent classloader first*, and loads the class *only if* the
parent classloader doesn't load/have it. In other words, any classpath the URLClassLoader
owns is effectively *appended*, not prepended. That's precisely why ApplicationClassLoader
inverts that order to create isolation.

Could you write a simple test program to verify this behavior? I'm pretty sure you'll find
that your classpath will still be shadowed by the system classpath.

Also, as for using the ApplicationClassLoader, it shouldn't be too difficult. You pass in
{{URL[]}} to the URLClassLoader too, so that's common. You can simply pass in the classloader
of the calling class as the parent classloader. Also, you can simply pass null for the system
classes, in which case the sensible default will be used.

If it helps anyway, we could introduce a simpler constructor like the following:
public ApplicationClassLoader(URL[] classpath) {
  this(classpath, getClass().getClassLoader(), null);

> Enable aux services to have their own custom classpath/jar file
> ---------------------------------------------------------------
>                 Key: YARN-4577
>                 URL: https://issues.apache.org/jira/browse/YARN-4577
>             Project: Hadoop YARN
>          Issue Type: Improvement
>    Affects Versions: 2.8.0
>            Reporter: Xuan Gong
>            Assignee: Xuan Gong
>         Attachments: YARN-4577.1.patch, YARN-4577.2.patch
> Right now, users have to add their jars to the NM classpath directly, thus put them on
the system classloader. But if multiple versions of the plugin are present on the classpath,
there is no control over which version actually gets loaded. Or if there are any conflicts
between the dependencies introduced by the auxiliary service and the NM itself, they can break
the NM, the auxiliary service, or both.
> The solution could be: to instantiate aux services using a classloader that is different
from the system classloader.

This message was sent by Atlassian JIRA

View raw message