hadoop-common-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Sharad Agarwal (JIRA)" <j...@apache.org>
Subject [jira] Updated: (HADOOP-3570) Including user specified jar files in the client side classpath path in Hadoop 0.17 streaming
Date Fri, 05 Sep 2008 06:47:44 GMT

     [ https://issues.apache.org/jira/browse/HADOOP-3570?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Sharad Agarwal updated HADOOP-3570:
-----------------------------------

    Attachment: 3570_v2.patch

Setting the classpath in JobClient.configureCommandLineOptions. I think it would be the better
place than GenericOptionsParser. The classloader is set in the thread context as well as in
the conf. 
Also, modified an existing test case to test this.

> Including user specified jar files in the client side classpath path in Hadoop 0.17 streaming
> ---------------------------------------------------------------------------------------------
>
>                 Key: HADOOP-3570
>                 URL: https://issues.apache.org/jira/browse/HADOOP-3570
>             Project: Hadoop Core
>          Issue Type: Bug
>          Components: contrib/streaming
>    Affects Versions: 0.17.0
>            Reporter: Suhas Gogate
>            Assignee: Mahadev konar
>             Fix For: 0.19.0
>
>         Attachments: 3570_v1.patch, 3570_v2.patch
>
>
> Hadoop 0.17 streaming allows specifying multiple user specific jar files on the command
line along with the hadoop-streaming.jar. These user specific jar files are not automatically
 added on the client side class path when hadoop runs the streaming job
> For example, in the following streaming job <user-app.jar> is not automatically
included in client side path before executing the streaming job
>  
> hadoop  jar -libjars <user-app.jar> $HADOOP_HOME/hadoop-streaming.jar \
>         -input </input-data-path> \
>         -output </output data path> \
>         -inputformat  <Input-format> \
>         -outputformat <OutputFormat> \
>         -mapper /bin/cat -reducer /bin/cat \
>         -additionalconfspec  <xxxx> \
>         -jobconf mapred.reduce.tasks=3\
> Workaround:
> Current work around is to set these user specific jar files in
> HADOOP_CLASSPATH environment variable on the client side before running above example.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


Mime
View raw message