hadoop-mapreduce-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Philip Zeyliger <phi...@cloudera.com>
Subject Re: Add user jars to mapreduce
Date Thu, 31 Dec 2009 23:25:35 GMT
You should use the DistributedCache.
http://hadoop.apache.org/common/docs/current/mapred_tutorial.html#DistributedCache



On Thu, Dec 31, 2009 at 3:21 PM, Eric Yang <eyang@yahoo-inc.com> wrote:

> Hi,
>
> I have a mapreduce program embeded in a java application, and I am trying
> to
> load additional jar files as add-on for the execution of my map reduce job.
> My program works like this:
>
> JobConf conf = new JobConf(new Configuration(), Demux.class);
> conf.setBoolean("mapred.used.genericoptionsparser",true);
>
> args[x]="-libjars"
> args[x+1]="/path/to/addon.jar";
>
> int res = ToolRunner.run(conf, new Demux(), args);
>
> When the mapreduce job is running, the task failed because it is unable to
> find the additional classes inside addon.jar.  If I extract
> /mapredsystem/hadoop/mapredsystem/job_200912171752_25474/job.jar, it looks
> like only the main jar file was in the job.jar.  Does the addon.jar needs
> to
> be preloaded into all task tracker node?  Could the addon.jar get uploaded
> via the client?  If yes, how to do this properly?
>
> I am using Hadoop 0.20.0 rc7 from yahoo with capacity scheduler.  Thanks in
> advance.
>
> Regards,
> Eric
>
>

Mime
View raw message