hadoop-common-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Vladimir Klimontovich (JIRA)" <j...@apache.org>
Subject [jira] Created: (HADOOP-6140) addArchiveToClassPath doesn't work in 0.18.x branch
Date Fri, 10 Jul 2009 17:11:15 GMT
addArchiveToClassPath doesn't work in 0.18.x branch
---------------------------------------------------

                 Key: HADOOP-6140
                 URL: https://issues.apache.org/jira/browse/HADOOP-6140
             Project: Hadoop Common
          Issue Type: Bug
          Components: fs
    Affects Versions: 0.18.3
            Reporter: Vladimir Klimontovich
            Priority: Minor


addArchiveToClassPath is a method of DistributedCache class. It should be called before running
a task. It accepts path to a jar file on a DFS. After it
this method should put this jar file on sitribuuted cache and than add this file to classpath
to each map/reduce process on job tracker.

This method didn't work.

Bug 1:

addArchiveToClassPath adds DFS-path to archive to mapred.job.classpath.archives property.
It uses System.getProperty("path.separator") as delimiter of multiple path.

getFileClassPaths that is called from TaskRunner uses splits mapred.job.classpath.archives
using System.getProperty("path.separator").

In unix systems System.getProperty("path.separator") equals to ":". DFS-path urls is hdfs://host:port/path.
It means that a result of split will be
[ hdfs,//host,port/path].

Suggested solution: use "," instead of  

Bug 2:

in TaskRunner there is an algorithm that looks for correspondence between DFS paths and local
paths in distributed cache. 
It compares

   if (archives[i].getPath().equals(
                                               archiveClasspaths[j].toString())){

instead of

    if (archives[i].toString().equals(
                                               archiveClasspaths[j].toString())) 


-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


Mime
View raw message