apex-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "ASF GitHub Bot (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (APEXCORE-304) Ability to add jars to classpath in populateDAG
Date Fri, 29 Apr 2016 05:18:12 GMT

    [ https://issues.apache.org/jira/browse/APEXCORE-304?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15263558#comment-15263558

ASF GitHub Bot commented on APEXCORE-304:

Github user chinmaykolhatkar commented on a diff in the pull request:

    --- Diff: engine/src/main/java/com/datatorrent/stram/plan/logical/LogicalPlan.java ---
    @@ -2386,7 +2381,7 @@ public static void write(DAG dag, OutputStream os) throws IOException
       public static LogicalPlan read(InputStream is) throws IOException, ClassNotFoundException
    -    return (LogicalPlan)new ObjectInputStream(is).readObject();
    +    return (LogicalPlan)new ClassLoaderObjectInputStream(Thread.currentThread().getContextClassLoader(),
    --- End diff --
    This is required for Local Mode case.
    Lets say a jar path is set to  LIBRARY_JARS and this jar contains a POJO that needs to
be set to TUPLE_CLASS attr. Now,  one can do that in populateDAG method, that would cause
a problem in  LocalMode as follows:
    1. populateDAG sets TUPLE_CLASS attr with new POJO class - Works.
    2. Local mode serializes the LogicalPlan - Works.
    3. Local mode deserialized the LogicalPlan - Fails.
    3rd step fails becuase The ObjectInputStream deserializer uses System ClassLoader which
does that have that POJO. ClassLoaderObjectInputStream is an overridden version of ObjectInputStream
which allows to set the custom classloader.
    By this time of callstack, Thread.currentThread().getClassLoader() will have a classloader
which is super of all.
    As previously, there was no functionality of LIBRARY_JARS, I guess this was not tested
and so did not surface as an issue.
    For cluster mode, there is no difference. i.e. There is only 1 classloader i.e. System.Classloader
and that has new jars as well.
    i.e. For cluster mode, Thread.currentThread().getClassLoader() = System classloader
    i.e. ClassLoaderObjectInputStream will behave as if ObjectInputStream.

> Ability to add jars to classpath in populateDAG
> -----------------------------------------------
>                 Key: APEXCORE-304
>                 URL: https://issues.apache.org/jira/browse/APEXCORE-304
>             Project: Apache Apex Core
>          Issue Type: Improvement
>            Reporter: Chinmay Kolhatkar
>            Assignee: Chinmay Kolhatkar
> This will have following functionality:
> 1) In populateDAG one would be allowed to add given local jar path to classpath of the
> 2) Optionally delete the given jar file after copying to HDFS is done.

This message was sent by Atlassian JIRA

View raw message