hadoop-common-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Sangjin Lee (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HADOOP-13070) classloading isolation improvements for cleaner and stricter dependencies
Date Sat, 28 May 2016 01:20:13 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-13070?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15305075#comment-15305075

Sangjin Lee commented on HADOOP-13070:

Thanks for the comments [~raviprak]! To answer your questions...

ApplicationClassLoader seems like its only being used by MR. I grepped in Tez and Spark source,
and didn't find any instances. Even if we were to do this only for MR, it would be incredibly
valuable. I feel it would also set a precedent / pattern that other frameworks can then leverage.
If you meant that the only usage is hadoop itself, I believe that's correct. Within hadoop,
there are 3 usages today: MR task class isolation, hadoop run jar class isolation, and more
recently the NM aux service class isolation. Since {{ApplicationClassLoader}} is part of the
public API, other frameworks can use it.

bq. If we were to focus on MR, do you know what are the common problematic conflicting dependencies?
Unfortunately there are many to choose from, and quite a few of the well-known ones fall into
the problem category. Some of the more famous ones include guava and jackson to name a couple.

But isolating class spaces has more benefits than simply preventing collisions. Since we're
afraid of breaking users, hadoop has been very slow/conservative in upgrading any libraries
it uses. As a result, we're stuck in the stone age for many of the libraries we use. Isolation
would give hadoop more freedom to upgrade its dependencies without worrying about impacting
users. That is of course provided that the isolation mode becomes the default, which may still
be some time away.

One alternative approach would be to start 2 JVMs for each MR-task: an MR-framework JVM and
an MR-task JVM. We would do all MR-framework specific work in the MR-framework JVM and send
raw Map-Reduce input key-value pairs over a socket and read output key value pairs over a
socket from the MR-task JVM. The MR specific code running in the MR-task JVM would then be
minimal and only needs to read over the socket and call the user code.
That is an interesting idea to solve this problem. I still worry about the performance implication
it has. Also, it still would not eliminate the problem entirely. As you pointed out, even
in that separate process you still need a minimal amount of hadoop code which then pulls in
the needed dependencies.

> classloading isolation improvements for cleaner and stricter dependencies
> -------------------------------------------------------------------------
>                 Key: HADOOP-13070
>                 URL: https://issues.apache.org/jira/browse/HADOOP-13070
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: util
>            Reporter: Sangjin Lee
>            Assignee: Sangjin Lee
>            Priority: Critical
>         Attachments: classloading-improvements-ideas-v.3.pdf, classloading-improvements-ideas.pdf,
> Related to HADOOP-11656, we would like to make a number of improvements in terms of classloading
isolation so that user-code can run safely without worrying about dependency collisions with
the Hadoop dependencies.
> By the same token, it should raised the quality of the user code and its specified classpath
so that users get clear signals if they specify incorrect classpaths.
> This will contain a proposal that will include several improvements some of which may
not be backward compatible. As such, it should be targeted to the next major revision of Hadoop.

This message was sent by Atlassian JIRA

To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org

View raw message