hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Vinod Kumar Vavilapalli (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (YARN-4577) Enable aux services to have their own custom classpath/jar file
Date Thu, 14 Apr 2016 05:57:25 GMT

    [ https://issues.apache.org/jira/browse/YARN-4577?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15240658#comment-15240658

Vinod Kumar Vavilapalli commented on YARN-4577:

Just caught up with the wall of comments.

+1 in general for the ApplicationClassloader based solution. Aux-services was always a hack
since Chris Douglas and I originally introduced it, the better solution is to move these services
as first-class apps on top of YARN, but we are where we are.

bq. For example, if the aux service code depends on another class property (owned by the aux
service) in the configuration, that will be invoked via Configuration.getClass(), and it will
still use the system classloader to load that class. Then it's very likely that you'll get
a ClassNotFoundException.
Sangjin, you may be missing one important thing here - unlike in the MapReduce case, there
is no shared Configuration object between NodeManager and the specific aux-service implementation
here. We simply do not pass in any configuration anywhere as part of the AuxService APIs -
so this entire thread of reasoning about getClass() is no long a problem? If needed, we can
document advising against adding Conf as part of future API changes.

bq. The thread context classloader represents another similar problem. The moment the aux
service code hits a code path that does Class.forName() that loads classes via the thread
context classloader, and it needs to load an aux service-related class (that is not present
in the main NM classpath), you will get a ClassNotFoundException.
In addition to wrapping aux-service API calls under a class-loader, wouldn't it suffice to
simply have NM make all aux-services API calls in a separate thread whose ContextClassLoader
is changed to be another custom one that resolves both System classes as well as AuxServices

> Enable aux services to have their own custom classpath/jar file
> ---------------------------------------------------------------
>                 Key: YARN-4577
>                 URL: https://issues.apache.org/jira/browse/YARN-4577
>             Project: Hadoop YARN
>          Issue Type: Improvement
>    Affects Versions: 2.8.0
>            Reporter: Xuan Gong
>            Assignee: Xuan Gong
>         Attachments: YARN-4577.1.patch, YARN-4577.2.patch, YARN-4577.20160119.1.patch,
YARN-4577.20160204.patch, YARN-4577.3.patch, YARN-4577.3.rebase.patch, YARN-4577.4.patch
> Right now, users have to add their jars to the NM classpath directly, thus put them on
the system classloader. But if multiple versions of the plugin are present on the classpath,
there is no control over which version actually gets loaded. Or if there are any conflicts
between the dependencies introduced by the auxiliary service and the NM itself, they can break
the NM, the auxiliary service, or both.
> The solution could be: to instantiate aux services using a classloader that is different
from the system classloader.

This message was sent by Atlassian JIRA

View raw message