hadoop-mapreduce-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Alejandro Abdelnur <t...@cloudera.com>
Subject Re: Problems adding JARs to distributed classpath in Hadoop 0.20.2
Date Mon, 30 May 2011 16:43:14 GMT

Now I get what you are trying to do.

My recommendation would be:

* Use a Java action to do all the stuff prior to starting your MR job
* Use a mapreduce action to start your MR job
* If you need to propagate properties from the Java action to the MR action
you can use the <capture-output> flag.

If you still want to start your MR job from your Java action, then your Java
action should do all the setup the MapReduceMain class does before starting
the MR job (this will ensure delegation tokens and distributed cache is
avail to your MR job).



On Mon, May 30, 2011 at 6:34 AM, John Armstrong <john.armstrong@ccri.com>wrote:

> On Fri, 27 May 2011 15:47:23 -0700, Alejandro Abdelnur <tucu@cloudera.com>
> wrote:
> > John,
> >
> > If you are using Oozie, dropping all the JARs your MR jobs needs in the
> > Oozie WF lib/ directory should suffice. Oozie will make sure all those
> JARs
> > are in the distributed cache.
> That doesn't seem to work.  I have this JAR in the WF /lib/ directory
> because the Java job that launches the MR job needs it.  And yes, it's in
> the distributed cache for the wrapper MR job that Oozie uses to remotely
> run the Java job.  The problem is it's not available for the MR job that
> the Java job launches.
> Thanks, though, for the suggestion.

View raw message