hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jean-Daniel Cryans <jdcry...@apache.org>
Subject Re: Implementing TableMap interface
Date Fri, 24 Jul 2009 13:48:55 GMT
This is not a very java-way to do it. For example if you have let's
say 20 mapreduce jobs and you keep them in a version control system,
you will have to copy over the classes every time they change into the
hbase path... is that a good practice?

The exception you get says "wrong name", a google search will tell you
how to solve that.

So yes I say that you must provide your own jar, this is how it works
actually. Please have a look at the MR tutorial in the Hadoop doc
http://hadoop.apache.org/common/docs/r0.19.1/mapred_tutorial.html

If you only want to run your job in local mode and run it by bin/hbase
then you should put your jar either in the lib folder or in the HBase
classpath in conf/hbase-env.sh if you want to keep it somewhere else.

Hope this helps,

J-D

On Fri, Jul 24, 2009 at 9:29 AM, bharath
vissapragada<bharathvissapragada1990@gmail.com> wrote:
> Actually u might have seen that i have implemented the TableMap interface
> using MyTableMap class so that map can return <Text,Text> pairs .. i have
> kept the package name to be so and so .., otherwise it is not recognising
> TableMap interface and throwing an error .. such a package name implies that
> hbase will find the the class from <packagename as a path>/MyTableMap.class
> ... so what you say is should we create such a jar and provide it while
> running the program?
>
> Initially what i did was .. i added the class file to that path in hbase-jar
> and the program compiled succesfully ...
> but while running the class file using "bin/hbase <MRprogram>" it threw an
> error as follows ...
>
> Exception in thread "main" java.lang.NoClassDefFoundError: MyTableMap (wrong
> name: org/apache/hadoop/hbase/mapred/MyTableMap)
>    at java.lang.ClassLoader.defineClass1(Native Method)
>    at java.lang.ClassLoader.defineClass(ClassLoader.java:637)
>    at
> java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)
>    at java.net.URLClassLoader.defineClass(URLClassLoader.java:277)
>    at java.net.URLClassLoader.access$000(URLClassLoader.java:73)
>    at java.net.URLClassLoader$1.run(URLClassLoader.java:212)
>    at java.security.AccessController.doPrivileged(Native Method)
>    at java.net.URLClassLoader.findClass(URLClassLoader.java:205)
>    at java.lang.ClassLoader.loadClass(ClassLoader.java:323)
>    at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:294)
>    at java.lang.ClassLoader.loadClass(ClassLoader.java:268)
>    at java.lang.ClassLoader.loadClassInternal(ClassLoader.java:336)
> Could not find the main class: MyTableMap. Program will exit.
>
> Hoping for a reply!
>
> 2009/7/24 Jean-Daniel Cryans <jdcryans@apache.org>
>
>> I really wonder why you would want to do that... Because you don't
>> want to import the classes from that package? Anyways you don't have
>> to be in that jar to be in that package, it should still resolve the
>> full name of your class as long as it is somewhere in the Hadoop
>> classpath. Thus, you have to add the HBase jar in Hadoop classpath and
>> when you run your job you have to provide your own jar.
>>
>> I hope this helps.
>>
>> J-D
>>
>> 2009/7/24 john smith <js1987.smith@gmail.com>:
>> > Thanks for replying Daniel,
>> >
>> > The link you have provided gives the  info about adding a new jar to the
>> > classpath .. Thats fine ...
>> > Suppose consider the above program written by Bharath ..
>> >
>> > It contains ...
>> >
>> > package "org.apache.hadoop.hbase.mapred" ..
>> >
>> > This means that the new class should be in that jar in that particular
>> > folder ..  I think this is different to adding a new jar to classpath ..
>> >
>> > any comments?
>> >
>> > 2009/7/24 Jean-Daniel Cryans <jdcryans@apache.org>
>> >
>> >> Please see the doc
>> >>
>> >>
>> http://hadoop.apache.org/hbase/docs/current/api/org/apache/hadoop/hbase/mapred/package-summary.html#classpath
>> >>
>> >> J-D
>> >>
>> >> 2009/7/24 john smith <js1987.smith@gmail.com>:
>> >> > Yes i too have the same problem .. Can anyone tell me in detail how
to
>> >> > add new classes to the existing hbase jar or do we have a different
>> >> > method to include our own classes in the program ..
>> >> >
>> >> > On 7/24/09, bharath vissapragada <bharathvissapragada1990@gmail.com>
>> >> wrote:
>> >> >> Thanks it worked fine .. Do i need to update hbase-x.x.jar ?? or
is
>> >> there
>> >> >> some other procedure to use it in my program ... because when i
>> updated
>> >> >> hbase-x.x.jar .. it gave me NoClassDefFoundError while running
my
>> >> program!!
>> >> >>
>> >> >>
>> >> >>
>> >> >> 2009/7/24 Doğacan Güney <dogacan@gmail.com>
>> >> >>
>> >> >>> n Jul 24, 2009, at 1:00 PM, bharath vissapragada wrote:
>> >> >>>
>> >> >>>  Hi all,
>> >> >>>>
>> >> >>>> I wanted to implement TableMap interface so that "map"
function can
>> >> emit
>> >> >>>> <Text,Text>.. I wrote the code as follows
>> >> >>>>
>> >> >>>>
>> >>
>> -------------------------------------------------------------------------------------------------------
>> >> >>>> package org.apache.hadoop.hbase.mapred;
>> >> >>>>
>> >> >>>> import java.io.IOException;
>> >> >>>> import org.apache.hadoop.hbase.io.ImmutableBytesWritable;
>> >> >>>> import org.apache.hadoop.hbase.io.RowResult;
>> >> >>>> import org.apache.hadoop.io.Text;
>> >> >>>> import org.apache.hadoop.mapred.JobConf;
>> >> >>>> import org.apache.hadoop.mapred.MapReduceBase;
>> >> >>>> import org.apache.hadoop.mapred.OutputCollector;
>> >> >>>> import org.apache.hadoop.mapred.Reporter;
>> >> >>>>
>> >> >>>> public class MyTableMap
>> >> >>>> extends MapReduceBase
>> >> >>>> implements TableMap<ImmutableBytesWritable, RowResult>
{
>> >> >>>>
>> >> >>>>  /** constructor */
>> >> >>>>  public MyTableMap() {
>> >> >>>>   super();
>> >> >>>>  }
>> >> >>>> @SuppressWarnings("unchecked")
>> >> >>>>  public static void initJob(String table, String columns,
>> >> >>>>   Class<? extends TableMap> mapper, JobConf job)
{
>> >> >>>>   TableMapReduceUtil.initTableMapJob(table, columns, mapper,
>> >> >>>>     Text.class,
>> >> >>>>     Text.class, job);
>> >> >>>>  }
>> >> >>>> public void map(ImmutableBytesWritable key, RowResult value,
>> >> >>>>     OutputCollector<Text,Text> output,
>> >> >>>>     @SuppressWarnings("unused") Reporter reporter) throws
>> IOException
>> >> {
>> >> >>>>   output.collect(new Text("ss"), new Text("pp"));
>> >> >>>>  }
>> >> >>>> }
>> >> >>>>
>> >> >>>> When i compile this code .. i get the following error ....
>> >> >>>>
>> >> >>>> MyTableMap.java:35: org.apache.hadoop.hbase.mapred.MyTableMap
is
>> not
>> >> >>>> abstract and does not override abstract method
>> >> >>>>
>> >> >>>>
>> >>
>> map(org.apache.hadoop.hbase.io.ImmutableBytesWritable,org.apache.hadoop.hbase.io.RowResult,org.apache.hadoop.mapred.OutputCollector<org.apache.hadoop.hbase.io.ImmutableBytesWritable,org.apache.hadoop.hbase.io.RowResult>,org.apache.hadoop.mapred.Reporter)
>> >> >>>> in org.apache.hadoop.mapred.Mapper
>> >> >>>> public class MyTableMap
>> >> >>>>      ^
>> >> >>>> Can anyone help me out!!
>> >> >>>>
>> >> >>>
>> >> >>> Your class should implement TableMap<Text, Text> not
TableMap<IBW,
>> RR>.
>> >> >>>
>> >> >>> --
>> >> >>> Doğacan Güney
>> >> >>
>> >> >
>> >>
>> >
>>
>

Mime
View raw message