hadoop-general mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Christopher L Tubbs II <ctubb...@gmail.com>
Subject mapreduce classpath issue
Date Sat, 25 Jul 2009 03:35:24 GMT
So, I've been having an issue running on Mac OS, with classpath for
MapReduce jobs on version 0.20.0

The basic idea is that I want to add a jar as a dependency to a
MapReduce job. I know I can add the jar to HADOOP_CLASSPATH, but I do
not have access to hadoop-env.xml (administrator restricted).

I have been using the "hadoop jar" command to execute a class with a
main() contained within my jar. My class extends Configured and
implements Tool. The main() calls ToolRunner.run() to launch the job,
and I specify -libjars on the command line, which I see from the source,
uses the GenericOptionsParser to strip the "-libjars x.jar,y.jar,z.jar"
from the arguments passed to my class. I then get errors that I can't
find classes contained within my library jars.

>From within run(), my debugging code demonstrates that the jar files are
readable, and I can getResource() to verify that the classes are
available in those jars. However, I still get errors, because it doesn't
seem that at any point, the JVM is using the contextClassLoader that the
resources were added with.

Perhaps this is a bug with Mac OS's JVM 1.6, or am I doing something
wrong? Everything works fine in linux with the identical setup.


View raw message