hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Rekha Joshi <rekha...@yahoo-inc.com>
Subject Re: -libjars doesn't work with MR job
Date Wed, 20 Jan 2010 06:17:29 GMT
Not sure what error you get and if it is suggestive, but attimes where you place the libjars
option can make a difference.You can try adding the jar to your HADOOP_CLASSPATH and then
executing?

Cheers,
/R


On 1/20/10 9:50 AM, "Victor Hsieh" <victorhsieh@gmail.com> wrote:

Hi,

I was trying to run a mapreduce job with some jars but failed.  It seems
that jars specified in command line -libjars was not shipped to mapreduce
worker together.

After digging into the code, I found that deprecated API and current are
different from -libjars behavior (also -files and -archives).  In deprecated
API, JobClient.runJob() will copy -libjars to DistributedCache (more
precisely, GenericOptionParser parses the -libjars, saving as "tmpjars" in
configuration, then JobClient upload tmpjars).  However, in current API, I
didn't see anything related (by grepping tmpjar or something in
hadoop-0.20.1/src/).

Is there any helper function or something in current API?  Or I need to do
it myself like what JobClient do?

Help appreciated.

Victor


Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message