apex-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From chinmaykolhatkar <...@git.apache.org>
Subject [GitHub] incubator-apex-core pull request: APEXCORE-304 Added support for l...
Date Fri, 29 Apr 2016 05:17:42 GMT
Github user chinmaykolhatkar commented on a diff in the pull request:

    --- Diff: engine/src/main/java/com/datatorrent/stram/plan/logical/LogicalPlan.java ---
    @@ -2386,7 +2381,7 @@ public static void write(DAG dag, OutputStream os) throws IOException
       public static LogicalPlan read(InputStream is) throws IOException, ClassNotFoundException
    -    return (LogicalPlan)new ObjectInputStream(is).readObject();
    +    return (LogicalPlan)new ClassLoaderObjectInputStream(Thread.currentThread().getContextClassLoader(),
    --- End diff --
    This is required for Local Mode case.
    Lets say a jar path is set to  LIBRARY_JARS and this jar contains a POJO that needs to
be set to TUPLE_CLASS attr. Now,  one can do that in populateDAG method, that would cause
a problem in  LocalMode as follows:
    1. populateDAG sets TUPLE_CLASS attr with new POJO class - Works.
    2. Local mode serializes the LogicalPlan - Works.
    3. Local mode deserialized the LogicalPlan - Fails.
    3rd step fails becuase The ObjectInputStream deserializer uses System ClassLoader which
does that have that POJO. ClassLoaderObjectInputStream is an overridden version of ObjectInputStream
which allows to set the custom classloader.
    By this time of callstack, Thread.currentThread().getClassLoader() will have a classloader
which is super of all.
    As previously, there was no functionality of LIBRARY_JARS, I guess this was not tested
and so did not surface as an issue.
    For cluster mode, there is no difference. i.e. There is only 1 classloader i.e. System.Classloader
and that has new jars as well.
    i.e. For cluster mode, Thread.currentThread().getClassLoader() = System classloader
    i.e. ClassLoaderObjectInputStream will behave as if ObjectInputStream.

If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.

View raw message