hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Aaron Kimball <aa...@cloudera.com>
Subject Re: Running MapReduce without setJar
Date Tue, 07 Apr 2009 18:51:00 GMT
All the nodes in your Hadoop cluster need access to the class files for your
MapReduce job. The current mechanism that Hadoop has to distribute classes
and attach them to the classpath assumes they're in a JAR together. Thus,
merely specifying the names of mapper/reducer classes with setMapperClass(),
etc, isn't enough -- you need to actually deliver a jar containing those
classes to all your nodes. Since the mapper and reducer classes are separate
classes, you'd need to bundle those .class files together somehow. JAR is
the standard way to do this, so that's what Hadoop supports.

If you're running in fully-local mode (e.g., with
jobConf.set("mapred.job.tracker", "local")), then no jar is needed since
it's all running inside the original process space.

- Aaron

On Thu, Apr 2, 2009 at 1:13 PM, Farhan Husain <russoue@gmail.com> wrote:

> I did all of them i.e. I used setMapClass, setReduceClass and new
> JobConf(MapReduceWork.class) but still it cannot run the job without a jar
> file. I understand the reason that it looks for those classes inside a jar
> but I think there should be some better way to find those classes without
> using a jar. But I am not sure whether it is possible at all.
> On Thu, Apr 2, 2009 at 2:56 PM, Rasit OZDAS <rasitozdas@gmail.com> wrote:
> > You can point to them by using
> > conf.setMapClass(..) and conf.setReduceClass(..)  - or something
> > similar, I don't have the source nearby.
> >
> > But something weird has happened to my code. It runs locally when I
> > start it as java process (tries to find input path locally). I'm now
> > using trunk, maybe something has changed with new version. With
> > version 0.19 it was fine.
> > Can somebody point out a clue?
> >
> > Rasit
> >
> --
> Mohammad Farhan Husain
> Research Assistant
> Department of Computer Science
> Erik Jonsson School of Engineering and Computer Science
> University of Texas at Dallas

  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message