hadoop-mapreduce-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Amareshwari Sri Ramadasu <amar...@yahoo-inc.com>
Subject Re: Add user jars to mapreduce
Date Mon, 04 Jan 2010 03:54:40 GMT



On 1/1/10 4:51 AM, "Eric Yang" <eyang@yahoo-inc.com> wrote:

Hi,

I have a mapreduce program embeded in a java application, and I am trying to
load additional jar files as add-on for the execution of my map reduce job.
My program works like this:

JobConf conf = new JobConf(new Configuration(), Demux.class);
conf.setBoolean("mapred.used.genericoptionsparser",true);

args[x]="-libjars"
args[x+1]="/path/to/addon.jar";

int res = ToolRunner.run(conf, new Demux(), args);

When the mapreduce job is running, the task failed because it is unable to
find the additional classes inside addon.jar.  If I extract
/mapredsystem/hadoop/mapredsystem/job_200912171752_25474/job.jar, it looks
like only the main jar file was in the job.jar.  Does the addon.jar needs to
be preloaded into all task tracker node?  Could the addon.jar get uploaded
via the client?  If yes, how to do this properly?

Your above code should work properly. Added jars will not be part of job.jar.
The addon.jar will be uploaded by the client, the added libjars will be added to classpath
using DistributedCache.
So, above code should work unless your path to jar or the jar itself was wrong.

I am using Hadoop 0.20.0 rc7 from yahoo with capacity scheduler.  Thanks in
advance.

Regards,
Eric



Mime
View raw message