hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Sangjin Lee (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (YARN-4577) Enable aux services to have their own custom classpath/jar file
Date Wed, 10 Feb 2016 02:03:18 GMT

    [ https://issues.apache.org/jira/browse/YARN-4577?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15140203#comment-15140203

Sangjin Lee commented on YARN-4577:

Thanks for updating the patch [~xgong].

It looks like the secondary classloading pattern has still not been addressed. I'm referring
to setting the created classloader onto the configuration as well as thread context classloader.
It is not sufficient to simply load the main aux service class using that classloader. That
works for the cases where other dependent classes are loaded via normal class references from
it, but does nothing to handle classloading via {{Configuration.getClass()}} or reflection
using the thread context classloader. If we do not address them, we will get fatal errors
the moment an aux service does those types of classloading, each of which will become a bug.
This is definitely a requirement IMO.

Fortunately we have a fairly well-defined set of call points that call into the aux services.
We can surround them with setting and unsetting the configuration classloader as well as thread
context classloader (see the comments above for how it is done). It is not exact, but it is
certainly necessary. Let me know if you have any questions.

Are we supporting hdfs paths as part of the aux classpaths? I thought that you mentioned that
it does not have to be done as part of this JIRA. If that is the case, why do we still need
to set the URL stream handler factory? The JVM's URL stream handler factory is capable of
handling all local paths.

Assuming we don't need to support hdfs paths, can't we simply rely on {{ApplicationClassLoader}}
to construct the URLs from the classpath? Is there a reason we need to replicate the {{constructUrlsFromClasspath()}}?
It would be good if we can rely on the common implementation, and improve it if there is a
missing piece.

> Enable aux services to have their own custom classpath/jar file
> ---------------------------------------------------------------
>                 Key: YARN-4577
>                 URL: https://issues.apache.org/jira/browse/YARN-4577
>             Project: Hadoop YARN
>          Issue Type: Improvement
>    Affects Versions: 2.8.0
>            Reporter: Xuan Gong
>            Assignee: Xuan Gong
>         Attachments: YARN-4577.1.patch, YARN-4577.2.patch, YARN-4577.20160119.1.patch,
YARN-4577.20160204.patch, YARN-4577.3.patch, YARN-4577.3.rebase.patch, YARN-4577.4.patch
> Right now, users have to add their jars to the NM classpath directly, thus put them on
the system classloader. But if multiple versions of the plugin are present on the classpath,
there is no control over which version actually gets loaded. Or if there are any conflicts
between the dependencies introduced by the auxiliary service and the NM itself, they can break
the NM, the auxiliary service, or both.
> The solution could be: to instantiate aux services using a classloader that is different
from the system classloader.

This message was sent by Atlassian JIRA

View raw message