hadoop-mapreduce-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Robert Joseph Evans (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (MAPREDUCE-4740) only .jars can be added to the Distributed Cache classpath
Date Mon, 22 Oct 2012 16:16:14 GMT

     [ https://issues.apache.org/jira/browse/MAPREDUCE-4740?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel

Robert Joseph Evans updated MAPREDUCE-4740:

    Attachment: MR-4740.txt

Updated version that allows for a cache archive to have a symlink name, and it to also be
added to the class path.  This does not update the API to set this that will still not work
if there is a #symlink in the archive being added to the classpath, but internally if the
confs are set manually it will work, and the unit test verifies it.
> only .jars can be added to the Distributed Cache classpath
> ----------------------------------------------------------
>                 Key: MAPREDUCE-4740
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4740
>             Project: Hadoop Map/Reduce
>          Issue Type: Bug
>          Components: mrv2
>    Affects Versions: 0.23.3, 2.0.2-alpha
>            Reporter: Robert Joseph Evans
>            Assignee: Robert Joseph Evans
>            Priority: Blocker
>         Attachments: MR-4740.txt, MR-4740.txt
> Koji tracked down this one.
> {noformat}
> $ cat mycat.sh
> #!/bin/sh
> cat >& /dev/null
> $JAVA_HOME/bin/jinfo $PPID | grep java.class.path
> export | grep CLASSPATH
> ls -l 
> $ myfile=/user/me/myclasses.zip; yarn jar
> hadoop-streaming.jar
> -Dmapreduce.job.cache.archives=hdfs:///${myfile}
> -Dmapreduce.job.classpath.archives=${myfile} -input in.txt -output out -reducer NONE
-mapper mycat.sh  -file mycat.sh
> {noformat}
> So, cachearchive like class.zip or class.tar.gz were never set as part of the
> classpath even though they were properly set by DistributedCache.addArchiveToClassPath.
> It looks like we are parsing the classpath out of the configs, but then throwing that
away.  It looks simple enough to add them in the correct place.

This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

View raw message