hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Sangjin Lee (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (YARN-4577) Enable aux services to have their own custom classpath/jar file
Date Wed, 13 Jan 2016 17:15:39 GMT

    [ https://issues.apache.org/jira/browse/YARN-4577?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15096584#comment-15096584

Sangjin Lee commented on YARN-4577:

Hi [~xgong], it would be great if you could describe precisely the behavior of the classloading
you desire in this solution. Then, we could discuss it a little better.

For your reference, here is what the ApplicationClassLoader does.
- isolates the classloading and the classpath of the "application" from the hadoop stack
- while this classloader is in place, it tries to load classes *first* from the user classpath
(as opposed to system classpath), and if not found tries to load it from the system classpath/classloader
- there are *"system classes"* (mainly hadoop classes and JDK classes) that are always loaded
only by the system classloader to ensure consistency
- outside the context of this classloader, the hadoop code does not see the user classpath
at all, and uses only the system classloader
- each application classloader is isolated from one another (obviously)
- when the application classloader is in scope, it gets set onto the Configuration as well
as the current thread's context classloader to ensure consistency for reflection-based classloading

This is essentially the same behavior as the webapp classloader of servlet engine implementations
(Tomcat, Jetty, etc.).

>From the description and the inferred behavior from the provided patch, I didn't see much
that the application classloader cannot work for this use case. The desire here is to see
if we can have a single generic solution that can address all the needs for isolated classloading,
rather than creating more solutions as new use cases arise. If there are some things that
are not addressed by the current application classloader implementation, we could consider
modifying it to make it wider in scope. There are some HADOOP JIRAs filed (see HADOOP-11656
for example) to adopt a single mechanism for classloading isolation and make it more formal
and stricter .


> Enable aux services to have their own custom classpath/jar file
> ---------------------------------------------------------------
>                 Key: YARN-4577
>                 URL: https://issues.apache.org/jira/browse/YARN-4577
>             Project: Hadoop YARN
>          Issue Type: Improvement
>    Affects Versions: 2.8.0
>            Reporter: Xuan Gong
>            Assignee: Xuan Gong
>         Attachments: YARN-4577.1.patch, YARN-4577.2.patch
> Right now, users have to add their jars to the NM classpath directly, thus put them on
the system classloader. But if multiple versions of the plugin are present on the classpath,
there is no control over which version actually gets loaded. Or if there are any conflicts
between the dependencies introduced by the auxiliary service and the NM itself, they can break
the NM, the auxiliary service, or both.
> The solution could be: to instantiate aux services using a classloader that is different
from the system classloader.

This message was sent by Atlassian JIRA

View raw message