hadoop-mapreduce-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Eric Yang <ey...@yahoo-inc.com>
Subject Add user jars to mapreduce
Date Thu, 31 Dec 2009 23:21:33 GMT
Hi,

I have a mapreduce program embeded in a java application, and I am trying to
load additional jar files as add-on for the execution of my map reduce job.
My program works like this:

JobConf conf = new JobConf(new Configuration(), Demux.class);
conf.setBoolean("mapred.used.genericoptionsparser",true);

args[x]="-libjars"
args[x+1]="/path/to/addon.jar";

int res = ToolRunner.run(conf, new Demux(), args);

When the mapreduce job is running, the task failed because it is unable to
find the additional classes inside addon.jar.  If I extract
/mapredsystem/hadoop/mapredsystem/job_200912171752_25474/job.jar, it looks
like only the main jar file was in the job.jar.  Does the addon.jar needs to
be preloaded into all task tracker node?  Could the addon.jar get uploaded
via the client?  If yes, how to do this properly?

I am using Hadoop 0.20.0 rc7 from yahoo with capacity scheduler.  Thanks in
advance.

Regards,
Eric


Mime
View raw message