hadoop-mapreduce-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Harsh J <ha...@cloudera.com>
Subject Re: Including external libraries in my job.
Date Tue, 03 May 2011 14:01:54 GMT

Am moving this to hbase-user, since its more relevant to HBase here
than MR's typical job submissions.

My reply below:

On Tue, May 3, 2011 at 7:12 PM, Niels Basjes <Niels@basjes.nl> wrote:
> Hi,
> I've written my first very simple job that does something with hbase.
> Now when I try to submit my jar in my cluster I get this:
> [nbasjes@master ~/src/catalogloader/run]$ hadoop jar
> catalogloader-1.0-SNAPSHOT.jar nl.basjes.catalogloader.Loader
> /user/nbasjes/Minicatalog.xml
> Exception in thread "main" java.lang.NoClassDefFoundError:
> org/apache/hadoop/hbase/HBaseConfiguration
>        at nl.basjes.catalogloader.Loader.main(Loader.java:156)
>        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> ...
> So what is the correct way of doing this?

The best way to write a Job Driver for HBase would be to use its
TableMapReduceUtil class to make it add dependent jars, prepare jobs
with a Scan, etc. [1].

Once your driver reflects the use of TableMapReduceUtil, simply do
(assuming HBase's bin/ is on PATH as well):
$ HADOOP_CLASSPATH=`hbase classpath` hadoop jar
nl.basjes.catalogloader.Loader /user/nbasjes/Minicatalog.xml

The "hbase classpath" command is magic for generating the proper
hbase-env.sh classpath out for use.

If you would still like to use -libjars to add in aux jars, make your
Driver use the GenericOptionsParser class [2]. Something like:

main(args) {
parser = new GenericOptionsParser(args);
conf = parser.getConfiguration();
rem_args = parser.getRemainingArgs();
// Do extra args processing if any..
// use 'conf' for your Job, not a new instance.

[1] - http://hbase.apache.org/apidocs/org/apache/hadoop/hbase/mapreduce/TableMapReduceUtil.html
[2] - http://hadoop.apache.org/common/docs/r0.20.2/api/org/apache/hadoop/util/GenericOptionsParser.html

HTH :)

Harsh J

View raw message