hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Guozhang Wang <wangg...@gmail.com>
Subject Question about avoiding reflection with isolated class loader
Date Thu, 28 May 2015 21:06:59 GMT

I have a question that is probably related to MAPREDUCE-1700 / 5751 / 5957
<https://issues.apache.org/jira/browse/SPARK-1870>. Basically I have also
encountered the issue that with separate classloaders while developing a
programming framework where I have to use reflection inside the application

To simply the question, let's say my framework code depends on a jar with
an older version and the application code depends on the same jar with a
newer version (for example Avro 1.4 and 1.7). And let's say that jar has a
class named CommonClass. I used a customized post-delegation class loader
like Hadoop does so that these two version can exist separately into
separate class loader. And if I do sth. like:

    public static void main(String arg[]) throws Exception {
        // Say this is framework code
        new CommonClass().printJarVersion();

        // Set the customized class loader as thread context class loader
        Thread thread = Thread.currentThread();
        ClassLoader oldClassLoader = thread.getContextClassLoader();
        File appJar = new File("/dir/to/app/class/path");
        URL[] classpath = new URL[] { appJar.toURI().toURL() };
        PostDelegationClassLoader newClassLoader = new

        try {
            // Say this is application code, like process().
            new CommonClass().printJarVersion();

        } finally {


It will print:

CommonClass: version 1
CommonClass: version 1
CommonClass: version 2

As one can see I have to use reflection to explicitly specify the
customized class loader if I want to create the class with the new version.
This is definitely bad for the users. I saw there are some discussions
around 1700 and am wondering how Hadoop resolved this issue? Any help is
greatly appreciated.

-- Guozhang

View raw message