hadoop-common-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Thomas Friol (JIRA)" <j...@apache.org>
Subject [jira] Commented: (HADOOP-509) initTasks method in the JobInProgress class does not correctly configure the classloader with the custom map/reduce jar
Date Thu, 07 Sep 2006 06:48:23 GMT
    [ http://issues.apache.org/jira/browse/HADOOP-509?page=comments#action_12433033 ] 
            
Thomas Friol commented on HADOOP-509:
-------------------------------------

Yes and it seems there is some code located in the IsolationRunner class which creates a classloader
provided a JobConf and a working directory.

> initTasks method in the JobInProgress class does not correctly configure the classloader
with the custom map/reduce jar
> -----------------------------------------------------------------------------------------------------------------------
>
>                 Key: HADOOP-509
>                 URL: http://issues.apache.org/jira/browse/HADOOP-509
>             Project: Hadoop
>          Issue Type: Bug
>          Components: mapred
>    Affects Versions: 0.5.0
>            Reporter: Thomas Friol
>
> In this part of code :
>         JobConf jd = new JobConf(localJobFile);
>         FileSystem fs = FileSystem.get(conf);
>         String ifClassName = jd.get("mapred.input.format.class");
>         InputFormat inputFormat;
>         if (ifClassName != null && localJarFile != null) {
>           try {
>             ClassLoader loader =
>               new URLClassLoader(new URL[]{ localFs.pathToFile(localJarFile).toURL()
});
>             Class inputFormatClass = Class.forName(ifClassName, true, loader);
>             inputFormat = (InputFormat)inputFormatClass.newInstance();
>           } catch (Exception e) {
>             throw new IOException(e.toString());
>           }
>         } else {
>           inputFormat = jd.getInputFormat();
>         }
> I think that the URLClassLoader should be added the URLs to all the jars contained in
the lib directory of the mapred jar. Or maybe, the jar should be unjarred and the classpath
should be updated as well as it is done into the TaskRunner class :
>       String jar = conf.getJar();
>       if (jar != null) {                      // if jar exists, it into workDir
>         RunJar.unJar(new File(jar), workDir);
>         File[] libs = new File(workDir, "lib").listFiles();
>         if (libs != null) {
>           for (int i = 0; i < libs.length; i++) {
>             classPath.append(sep);            // add libs from jar to classpath
>             classPath.append(libs[i]);
>           }
>         }
>         classPath.append(sep);
>         classPath.append(new File(workDir, "classes"));
>         classPath.append(sep);
>         classPath.append(workDir);
>       }
> Actually, it is not possible to load a custom InputFormat class. Any thoughts ?
> PS : This issue comes following a discussion on the issue https://issues.apache.org/jira/browse/HADOOP-366

-- 
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

Mime
View raw message