apex-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "ASF GitHub Bot (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (APEXCORE-304) Ability to add jars to classpath in populateDAG
Date Thu, 28 Jan 2016 13:10:39 GMT

    [ https://issues.apache.org/jira/browse/APEXCORE-304?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15121393#comment-15121393

ASF GitHub Bot commented on APEXCORE-304:

Github user chinmaykolhatkar commented on a diff in the pull request:

    --- Diff: engine/src/main/java/com/datatorrent/stram/StramLocalCluster.java ---
    @@ -318,6 +333,21 @@ public StramLocalCluster(LogicalPlan dag) throws IOException, ClassNotFoundExcep
    +  private void addExtraJarsToClasspath(Set<String> jars)
    +      throws InvocationTargetException, IllegalAccessException, MalformedURLException,
    +  {
    +    List<URL> jarUrls = new LinkedList<>();
    +    for (String jarPath : jars) {
    +      File file = new File(jarPath);
    +      URL url = file.toURI().toURL();
    +      jarUrls.add(url);
    +    }
    +    ClassLoader prevCl = Thread.currentThread().getContextClassLoader();
    +    ClassLoader urlCl = URLClassLoader.newInstance(jarUrls.toArray(new URL[jarUrls.size()]),
    +    Thread.currentThread().setContextClassLoader(urlCl);
    --- End diff --
    Thomas can answer this question better. But here is additional information as per my understanding:
    The place where Constructor of StramLocalCluster is called is a common point where both
code paths joins.
    But there is a slight difference in how the operators are loaded by these 2 paths.
    In LocalMode.runApp, the operator classes are already part of the System class loader.
This is because when ApplicationTest is launched the operators are already in classpath provided
by maven.
    StreamAppLauncher.runLocal is invoked by an application launched from dtcli (a java process).
dtcli do not have operators present in its system classpath.
    Hence, in StramAppLauncher.runLocal (dtcli -local), there is a context class loader created
which has information about operator classes. This context classloader is later used to load
the operator classes.

> Ability to add jars to classpath in populateDAG
> -----------------------------------------------
>                 Key: APEXCORE-304
>                 URL: https://issues.apache.org/jira/browse/APEXCORE-304
>             Project: Apache Apex Core
>          Issue Type: Improvement
>            Reporter: Chinmay Kolhatkar
>            Assignee: Chinmay Kolhatkar
> This will have following functionality:
> 1) In populateDAG one would be allowed to add given local jar path to classpath of the
> 2) Optionally delete the given jar file after copying to HDFS is done.

This message was sent by Atlassian JIRA

View raw message