apex-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From chinmaykolhatkar <...@git.apache.org>
Subject [GitHub] incubator-apex-core pull request: APEXCORE-304 Added support for c...
Date Wed, 27 Jan 2016 11:35:53 GMT
Github user chinmaykolhatkar commented on a diff in the pull request:

    --- Diff: engine/src/main/java/com/datatorrent/stram/StramLocalCluster.java ---
    @@ -318,6 +333,21 @@ public StramLocalCluster(LogicalPlan dag) throws IOException, ClassNotFoundExcep
    +  private void addExtraJarsToClasspath(Set<String> jars)
    +      throws InvocationTargetException, IllegalAccessException, MalformedURLException,
    +  {
    +    List<URL> jarUrls = new LinkedList<>();
    +    for (String jarPath : jars) {
    +      File file = new File(jarPath);
    +      URL url = file.toURI().toURL();
    +      jarUrls.add(url);
    +    }
    +    ClassLoader prevCl = Thread.currentThread().getContextClassLoader();
    +    ClassLoader urlCl = URLClassLoader.newInstance(jarUrls.toArray(new URL[jarUrls.size()]),
    +    Thread.currentThread().setContextClassLoader(urlCl);
    --- End diff --
    You're right. There was a little modification required.
    So here is how I've addressed things for various run modes:
    **1. For cluster mode:**
    The jar gets copied to HDFS lib location and the hdfs jar path is added to the classpath.
    This is how it was done originally in this pull request and is unchanged since.
    **2. For local mode via dtcli (-local):**
    Idea is to add the extra jars to same context loader as the one which loads operator classes.
Because of this, the thread that gets spawned for operator/container is set with a context
loader having both extra jars and the operator code. Because of this the extra jars are available
by all ways. I'm not sure to write a unit test for -local mode of dtcli.  But I wrote a sample
test application and ran in local mode with dtcli. The operator setup tried to load the classes
by following APIs and it ran successfully:
      *Class.forName(<class>, init, Thread.currentThread().getContextClassLoader())*
    **3. For local mode via LocalMode.runApp API (i.e. ApplicationTest):**
    I've added the unit test in which operator loads the class via following APIs successfully:
      *Class.forName(<class>, init, Thread.currentThread().getContextClassLoader())*
    Class.forName will not work here as Class.forName find classloader of caller class and
then tries to load the class from caller's loader or its parent. As the operators are already
loaded by System classloader, there is no clean way to modify the class loader to add another
path to it.
    If its URL class loader, then via reflection can be done. But not sure if its clean again.

    Hence, operator will have to use Thread context OR StramUtils.classForName method to load
the class. 

If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.

View raw message