Mailing-List: contact user-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@hadoop.apache.org
MIME-Version: 1.0
Date: Thu, 28 May 2015 14:06:59 -0700
Message-ID: 
 <CAHwHRrUBrfsL_ws5gBviLnbj875aRPb0eMkoC9DzNGiCqBRSkA@mail.gmail.com>
Subject: Question about avoiding reflection with isolated class loader
From: Guozhang Wang <wangguoz@gmail.com>
To: user@hadoop.apache.org
Content-Type: multipart/alternative; boundary=bcaec5171bc9878c3b05172abd37

--bcaec5171bc9878c3b05172abd37
Content-Type: text/plain; charset=UTF-8

Hi,

I have a question that is probably related to MAPREDUCE-1700 / 5751 / 5957
<https://issues.apache.org/jira/browse/SPARK-1870>. Basically I have also
encountered the issue that with separate classloaders while developing a
programming framework where I have to use reflection inside the application
code.

To simply the question, let's say my framework code depends on a jar with
an older version and the application code depends on the same jar with a
newer version (for example Avro 1.4 and 1.7). And let's say that jar has a
class named CommonClass. I used a customized post-delegation class loader
like Hadoop does so that these two version can exist separately into
separate class loader. And if I do sth. like:

    public static void main(String arg[]) throws Exception {
        // Say this is framework code
        new CommonClass().printJarVersion();

        // Set the customized class loader as thread context class loader
        Thread thread = Thread.currentThread();
        ClassLoader oldClassLoader = thread.getContextClassLoader();
        File appJar = new File("/dir/to/app/class/path");
        URL[] classpath = new URL[] { appJar.toURI().toURL() };
        PostDelegationClassLoader newClassLoader = new
PostDelegationClassLoader(classpath);
        thread.setContextClassLoader(newClassLoader);

        try {
            // Say this is application code, like process().
            new CommonClass().printJarVersion();

thread.getContextClassLoader().loadClass("CommonClass").newInstance().printJarVersion();
        } finally {
            thread.setContextClassLoader(oldClassLoader);
        }
    }

-----------------------

It will print:

CommonClass: version 1
CommonClass: version 1
CommonClass: version 2

As one can see I have to use reflection to explicitly specify the
customized class loader if I want to create the class with the new version.
This is definitely bad for the users. I saw there are some discussions
around 1700 and am wondering how Hadoop resolved this issue? Any help is
greatly appreciated.

-- Guozhang

--bcaec5171bc9878c3b05172abd37
Content-Type: text/html; charset=UTF-8
Content-Transfer-Encoding: quoted-printable

<div dir=3D"ltr"><div><div>Hi,<br><br></div>I have a question that is proba=
bly related to MAPREDUCE-1700 / 5751 / 5957<a class=3D"" href=3D"https://is=
sues.apache.org/jira/browse/SPARK-1870" id=3D"key-val" rel=3D"12715034"></a=
>.
 Basically I have also encountered the issue that with separate=20
classloaders while developing a programming framework where I have to=20
use reflection inside the application code.<br><br></div><div>To simply=20
the question, let&#39;s say my framework code depends on a jar with an olde=
r
 version and the application code depends on the same jar with a newer=20
version (for example Avro 1.4 and 1.7). And let&#39;s say that jar has a cl=
ass named CommonClass. I used a=20
customized post-delegation class loader like Hadoop does so that these two =
version can=20
exist separately into separate class loader. And if I do sth. like:<br><br>=
=C2=A0=C2=A0=C2=A0 public static void main(String arg[]) throws Exception {=
<br></div><div>=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 // Say this is fr=
amework code<br></div><div>=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 new C=
ommonClass().printJarVersion();<br></div><div><br></div><div>=C2=A0=C2=A0=
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 // Set the customized class loader as thread=
 context class loader<br></div><div>=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=
=A0 Thread thread =3D Thread.currentThread();<br>=C2=A0=C2=A0=C2=A0=C2=A0=
=C2=A0=C2=A0=C2=A0 ClassLoader oldClassLoader =3D thread.getContextClassLoa=
der();<br>=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 File appJar =3D new Fi=
le(&quot;/dir/to/app/class/path&quot;);<br>=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=
=C2=A0=C2=A0 URL[] classpath =3D new URL[] { appJar.toURI().toURL() };<br>=
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 PostDelegationClassLoader newCla=
ssLoader =3D new PostDelegationClassLoader(classpath);<br>=C2=A0=C2=A0=C2=
=A0=C2=A0=C2=A0=C2=A0=C2=A0 thread.setContextClassLoader(newClassLoader);<b=
r><br>=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 try {<br></div><div>=C2=A0=
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 // Say this is=
 application code, like process().<br></div><div>=C2=A0=C2=A0=C2=A0=C2=A0=
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 new CommonClass().printJarVersio=
n();<br>=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 =
thread.getContextClassLoader().loadClass(&quot;CommonClass&quot;).newInstan=
ce().printJarVersion();<br>=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 } fin=
ally {<br>=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=
=A0 thread.setContextClassLoader(oldClassLoader);<br>=C2=A0=C2=A0=C2=A0=C2=
=A0=C2=A0=C2=A0=C2=A0 }<br>=C2=A0=C2=A0=C2=A0 }<br><br>--------------------=
---<br><br></div><div>It will print:<br><br></div><div>CommonClass: version=
 1<br>CommonClass: version 1<br>CommonClass: version 2<br></div><div><br></=
div><div>As
 one can see I have to use reflection to explicitly specify the=20
customized class loader if I want to create the class with the new=20
version. This is definitely bad for the users. I saw there are some discuss=
ions around 1700 and am wondering how Hadoop resolved this issue? Any help =
is greatly appreciated.<br><br></div><div><div><div><div dir=3D"ltr"><div>-=
- Guozhang<br></div></div>
</div></div></div></div>

--bcaec5171bc9878c3b05172abd37--