Mailing-List: contact user-help@hbase.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@hbase.apache.org
Received-SPF: pass (athena.apache.org: domain of a_portley@hotmail.com
 designates 65.55.111.80 as permitted sender)
Message-ID: <BLU120-W212142E3FF7FF5F04CAD5A983C0@phx.gbl>
Content-Type: multipart/alternative;
	boundary="_62ebe1d2-c04a-47f7-b87c-229c7f6b3553_"
From: Adam Portley <a_portley@hotmail.com>
To: <user@hbase.apache.org>
Subject: RE: ClassNotFoundException while running some HBase m/r jobs
Date: Sun, 21 Nov 2010 08:51:51 -0800
Importance: Normal
In-Reply-To: <AANLkTine=U-q=SNbxWpAFUA-G-cEu=Kw_2FhSzESaK8n@mail.gmail.com>
References: 
 <AANLkTimNAhywkx9jjd6_w=D94eLBw_qo4chGA+JzU4tn@mail.gmail.com>,<2DF32BA3-80E8-43F4-8F0E-293EFA6D5810@gmail.com>,<AANLkTimzE_9mGjKW3qZPgmhGDsCk4Y7Xb+jymABLoSqJ@mail.gmail.com>,<AANLkTim2h_nQRcid_ayBq_K+pnJFBqa5jOUCnYV4NzWe@mail.gmail.com>,<AANLkTikqCuVk7rZziWdTw+AskZcETC6EcRTR_+tDUDyF@mail.gmail.com>,<AANLkTi=qzYbzktorijX82M7qRqK=+0cV166YWFFv0PY=@mail.gmail.com>,<AANLkTi=yMbmgFTsXBSgWssFSZ89K8eaP-dN_-+fPCS-O@mail.gmail.com>,<AANLkTinvatv=VyM8DVM7cHYQAGFUNTU_w1Ge7OzGA+TC@mail.gmail.com>,<AANLkTine=U-q=SNbxWpAFUA-G-cEu=Kw_2FhSzESaK8n@mail.gmail.com>
MIME-Version: 1.0

--_62ebe1d2-c04a-47f7-b87c-229c7f6b3553_
Content-Type: text/plain; charset="iso-8859-1"
Content-Transfer-Encoding: quoted-printable


If the jar is included in HADOOP_CLASSPATH in your config but you're still =
getting ClassNotFound=2C you might try printing out the map task environmen=
t.  You can do this easily by running a hadoop streaming job with mapper se=
t to be printenv or something similar to print the environment to the outpu=
t path.  If the environment does not match the config=2C it often means tha=
t the TaskTrackers were not restarted after the config was edited. =20
=20
> Date: Sun=2C 21 Nov 2010 21:48:14 +0530
> Subject: Re: ClassNotFoundException while running some HBase m/r jobs
> From: hsreekumar@clickable.com
> To: user@hbase.apache.org
>=20
> Hi Ted=2C
> I tried doing the same thing with ant. And it worked! Thanks guys!
> But what I have now is a 26 MB fat jar file=2C since I included all jars =
I
> was including the the classpathref (shown above in ant script). Is there =
any
> other way I can get it to work? What is the root cause of this problem? T=
his
> solution works but looks very unclean. Ideally=2C the jar files should ge=
t
> found=2C right? Or is this meant to be used like this by design?
>=20
> thanks=2C
> hari
>=20
> On Sun=2C Nov 21=2C 2010 at 8:55 PM=2C Ted Yu <yuzhihong@gmail.com> wrote=
:
>=20
> > We package hbase class files (using maven) into our jar. E.g.
> > [hadoop@us01-ciqps1-name01 sims]$ jar tvf lib/flow-3.0.0.54-294813.jar =
|
> > grep hbase
> > ...
> > 6743 Sat Jul 03 09:17:38 GMT 2010
> > org/apache/hadoop/hbase/thrift/ThriftUtilities.class
> > 24472 Sat Jul 03 09:17:38 GMT 2010
> > org/apache/hadoop/hbase/thrift/ThriftServer$HBaseHandler.class
> > 3897 Sat Jul 03 09:17:38 GMT 2010
> > org/apache/hadoop/hbase/thrift/ThriftServer.class
> > 565 Sat Jul 03 07:16:26 GMT 2010
> > org/apache/hadoop/hbase/TableNotFoundException.class
> > 2306 Sat Jul 03 07:16:26 GMT 2010
> > org/apache/hadoop/hbase/HStoreKey$StoreKeyComparator.class
> > 722 Sat Jul 03 07:16:22 GMT 2010
> > org/apache/hadoop/hbase/DoNotRetryIOException.class
> >
> > FYI
> >
> > On Sun=2C Nov 21=2C 2010 at 7:18 AM=2C Hari Sreekumar <hsreekumar@click=
able.com
> > >wrote:
> >
> > > Hi Ted=2C
> > >
> > > Sure.. I use this command:
> > > $HADOOP_HOME/bin/hadoop jar ~/MRJobs.jar BulkUpload /tmp/customerData=
.dat
> > >
> > > /tmp/customerData.dat is the argument (text file from which data is t=
o be
> > > uploaded) and BulkUpload is the class name.
> > >
> > > thanks=2C
> > > hari
> > >
> > > On Sun=2C Nov 21=2C 2010 at 8:35 PM=2C Ted Yu <yuzhihong@gmail.com> w=
rote:
> > >
> > > > Can you show us the command which you use to launch the M/R job ?
> > > >
> > > > Thanks
> > > >
> > > > On Sun=2C Nov 21=2C 2010 at 5:26 AM=2C Hari Sreekumar <
> > > hsreekumar@clickable.com
> > > > >wrote:
> > > >
> > > > > Hey Lars=2C
> > > > > You mean copying all required jar files to the lib/ folder in eac=
h
> > > jar?
> > > > > Is it worth the redundancy? I'll check if it works if I do that.
> > > > Currently=2C
> > > > > I am using ant to build my jar file with these instructions to
> > include
> > > > > files:
> > > > >
> > > > > <path id=3D"classpath">
> > > > > <fileset dir=3D"${lib.dir}" includes=3D"**/*.jar"/>
> > > > > <fileset dir=3D"${env.HADOOP_HOME}" includes=3D"*.jar"/>
> > > > > <fileset dir=3D"${env.HBASE_HOME}" includes=3D"*.jar"/>
> > > > > <fileset dir=3D"${env.HADOOP_HOME}/lib" includes=3D"**/*.jar"/>
> > > > > <fileset dir=3D"${env.HBASE_HOME}/lib" includes=3D"**/*.jar"/>
> > > > > </path>
> > > > >
> > > > > <target name=3D"compile" depends=3D"clean">
> > > > > <mkdir dir=3D"${build.dir}"/>
> > > > > <javac srcdir=3D"${src.dir}" destdir=3D"${build.dir}"
> > > > > classpathref=3D"classpath"/>
> > > > > <copy todir=3D"${build.dir}">
> > > > > <fileset dir=3D"${input.dir}" includes=3D"*.*"/>
> > > > > </copy>
> > > > > </target>
> > > > >
> > > > > I'll try copying all jars into the lib and including only the lib
> > > folder
> > > > > now.
> > > > >
> > > > > thanks=2C
> > > > > hari
> > > > >
> > > > > On Sun=2C Nov 21=2C 2010 at 5:32 PM=2C Lars George <lars.george@g=
mail.com>
> > > > > wrote:
> > > > >
> > > > > > Hi Hari=2C
> > > > > >
> > > > > > I would try the "fat" jar approach. It is much easier to mainta=
in
> > as
> > > > > > each job jar contains its required dependencies. Adding it to t=
he
> > > > > > nodes and config is becoming a maintenance nightmare very quick=
ly.
> > I
> > > > > > am personally using Maven to build my job jars and the Maven
> > > "Package"
> > > > > > plugin that has a custom package descriptor which - upon buildi=
ng
> > the
> > > > > > project - wraps everything up for me in one fell swoop.
> > > > > >
> > > > > > Lars
> > > > > >
> > > > > > On Sun=2C Nov 21=2C 2010 at 8:17 AM=2C Hari Sreekumar
> > > > > > <hsreekumar@clickable.com> wrote:
> > > > > > > Hi Lars=2C
> > > > > > >
> > > > > > > I tried copying conf to all nodes and copying jar=2C it is
> > > still
> > > > > > > giving the same error. Weird thing is that tasks on the maste=
r
> > node
> > > > are
> > > > > > also
> > > > > > > failing with the same error=2C even though all my files are
> > available
> > > > on
> > > > > > > master. I am sure I'm missing something basic here=2C but una=
ble to
> > > > > > pinpoint
> > > > > > > the exact problem.
> > > > > > >
> > > > > > > hari
> > > > > > >
> > > > > > > On Sun=2C Nov 21=2C 2010 at 3:11 AM=2C Lars George <
> > > lars.george@gmail.com>
> > > > > > wrote:
> > > > > > >
> > > > > > >> Hi Hari=2C
> > > > > > >>
> > > > > > >> This is most certainly a classpath issue. You either have to=
 add
> > > the
> > > > > jar
> > > > > > to
> > > > > > >> all TaskTracker servers and add it into the hadoop-env.sh in=
 the
> > > > > > >> HADOOP_CLASSPATH line (and copy it to all servers again *and=
*
> > > > restart
> > > > > > the
> > > > > > >> TaskTracker process!) or put the jar into the job jar into a
> > /lib
> > > > > > directory.
> > > > > > >>
> > > > > > >> Lars
> > > > > > >>
> > > > > > >> On Nov 20=2C 2010=2C at 22:33=2C Hari Sreekumar <
> > > hsreekumar@clickable.com
> > > > >
> > > > > > >> wrote:
> > > > > > >>
> > > > > > >> > Hi=2C
> > > > > > >> >
> > > > > > >> > I am getting this exception while running m/r jobs on HBas=
e:
> > > > > > >> >
> > > > > > >> > 10/11/21 02:53:01 INFO input.FileInputFormat: Total input
> > paths
> > > to
> > > > > > >> process :
> > > > > > >> > 1
> > > > > > >> > 10/11/21 02:53:01 INFO mapred.JobClient: Running job:
> > > > > > >> job_201011210240_0002
> > > > > > >> > 10/11/21 02:53:02 INFO mapred.JobClient: map 0% reduce 0%
> > > > > > >> > 10/11/21 02:53:08 INFO mapred.JobClient: Task Id :
> > > > > > >> > attempt_201011210240_0002_m_000036_0=2C Status : FAILED
> > > > > > >> > java.lang.RuntimeException: java.lang.ClassNotFoundExcepti=
on:
> > > > > > >> > org.apache.hadoop.hbase.mapreduce.TableOutputFormat
> > > > > > >> > at
> > > > > > >> >
> > > > > org.apache.hadoop.conf.Configuration.getClass(Configuration.java:=
809)
> > > > > > >> > at
> > > > > > >> >
> > > > > > >>
> > > > > >
> > > > >
> > > >
> > >
> > org.apache.hadoop.mapreduce.JobContext.getOutputFormatClass(JobContext.=
java:193)
> > > > > > >> > at
> > > org.apache.hadoop.mapred.Task.initialize(Task.java:413)
> > > > > > >> > at
> > org.apache.hadoop.mapred.MapTask.run(MapTask.java:288)
> > > > > > >> > at org.apache.hadoop.mapred.Child.main(Child.java:170)
> > > > > > >> > Caused by: java.lang.ClassNotFoundException:
> > > > > > >> > org.apache.hadoop.hbase.mapreduce.TableOutputFormat
> > > > > > >> > at
> > java.net.URLClassLoader$1.run(URLClassLoader.java:202)
> > > > > > >> > at java.security.AccessController.doPrivileged(Native
> > > > Method)
> > > > > > >> > at
> > > > java.net.URLClassLoader.findClass(URLClassLoader.java:190)
> > > > > > >> > at
> > java.lang.ClassLoader.loadClass(ClassLoader.java:307)
> > > > > > >> > at
> > > > > > sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
> > > > > > >> > at
> > java.lang.ClassLoader.loadClass(ClassLoader.java:248)
> > > > > > >> > at java.lang.Class.forName0(Native Method)
> > > > > > >> > at java.lang.Class.forName(Class.java:247)
> > > > > > >> > at
> > > > > > >> >
> > > > > > >>
> > > > > >
> > > > >
> > > >
> > >
> > org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:=
762)
> > > > > > >> > at
> > > > > > >> >
> > > > > org.apache.hadoop.conf.Configuration.getClass(Configuration.java:=
807)
> > > > > > >> > ... 4 more
> > > > > > >> >
> > > > > > >> > What could be the probable reasons for this? I have made s=
ure
> > > that
> > > > > > >> > hbase-0.20.6.jar=2C which contains this particular class=
=2C is
> > > > included
> > > > > in
> > > > > > >> the
> > > > > > >> > class path. In fact=2C if I run non-m/r jobs=2C it works f=
ine.
> > e.g=2C
> > > I
> > > > > ran
> > > > > > a
> > > > > > >> jar
> > > > > > >> > file successfully that uses HAdmin to create some tables. =
Here
> > > is
> > > > a
> > > > > > part
> > > > > > >> of
> > > > > > >> > the output from these jobs:
> > > > > > >> >
> > > > > > >> > 10/11/21 02:49:24 INFO zookeeper.ZooKeeper: Client
> > > > > > >> > environment:java.vendor=3DSun Microsystems Inc.
> > > > > > >> > 10/11/21 02:49:24 INFO zookeeper.ZooKeeper: Client
> > > > > > >> > environment:java.home=3D/usr/java/jdk1.6.0_22/jre
> > > > > > >> > 10/11/21 02:49:24 INFO zookeeper.ZooKeeper: Client
> > > > > > >> >
> > > > > > >>
> > > > > >
> > > > >
> > > >
> > >
> > environment:java.class.path=3D/opt/hadoop/bin/../conf:/usr/java/jdk1.6.=
0_22/lib/tools.jar:/opt/hadoop/bin/..
> > > > > > >> >
> > > > > > >>
> > > > > >
> > > > >
> > > >
> > >
> > :/opt/hadoop/bin/../hadoop-0.20.2-core.jar:/opt/hadoop/bin/../lib/commo=
ns-cli-1.2.jar:/opt/hadoop/bin/../lib/commons-codec-1.3.jar:/opt/hadoop/bin=
/../lib/com
> > > > > > >> >
> > > > > > >>
> > > > > >
> > > > >
> > > >
> > >
> > mons-el-1.0.jar:/opt/hadoop/bin/../lib/commons-httpclient-3.0.1.jar:/op=
t/hadoop/bin/../lib/commons-logging-1.0.4.jar:/opt/hadoop/bin/../lib/common=
s-logging-a
> > > > > > >> >
> > > > > > >>
> > > > > >
> > > > >
> > > >
> > >
> > pi-1.0.4.jar:/opt/hadoop/bin/../lib/commons-net-1.4.1.jar:/opt/hadoop/b=
in/../lib/core-3.1.1.jar:/opt/hadoop/bin/../lib/hsqldb-1.8.0.10.jar:/opt/ha=
doop/bin/..
> > > > > > >> >
> > > > > > >>
> > > > > >
> > > > >
> > > >
> > >
> > /lib/jasper-compiler-5.5.12.jar:/opt/hadoop/bin/../lib/jasper-runtime-5=
.5.12.jar:/opt/hadoop/bin/../lib/jets3t-0.6.1.jar:/opt/hadoop/bin/../lib/je=
tty-6.1.14.
> > > > > > >> >
> > > > > > >>
> > > > > >
> > > > >
> > > >
> > >
> > jar:/opt/hadoop/bin/../lib/jetty-util-6.1.14.jar:/opt/hadoop/bin/../lib=
/junit-3.8.1.jar:/opt/hadoop/bin/../lib/kfs-0.2.2.jar:/opt/hadoop/bin/../li=
b/log4j-1.2.15.jar:/opt/hadoop/bin/../lib/mockito-all-1.8.0.jar:/opt/hadoop=
/bin/../lib/oro-2.0.8.jar:/opt/hadoop/bin/../lib/servlet-api-2.5-6.1.14.jar=
:/opt/hadoop/bin/../lib/slf4j-api-1.4.3.jar:/opt/hadoop/bin/../lib/slf4j-lo=
g4j12-1.4.3.jar:/opt/hadoop/bin/../lib/xmlenc-0.52.jar:/opt/hadoop/bin/../l=
ib/jsp-2.1/jsp-2.1.jar:/opt/hadoop/bin/../lib/jsp-2.1/jsp-api-2.1.jar:/opt/=
hbase/hbase-0.20.6.jar:/opt/hbase/hbase-0.20.6-test.jar:/opt/hbase/conf:/op=
t/hbase/lib/zookeeper-3.2.2.jar
> > > > > > >> > 10/11/21 02:49:24 INFO zookeeper.ZooKeeper: Client
> > > > > > >> >
> > > > > > >>
> > > > > >
> > > > >
> > > >
> > >
> > environment:java.library.path=3D/opt/hadoop/bin/../lib/native/Linux-amd=
64-64
> > > > > > >> > 10/11/21 02:49:24 INFO zookeeper.ZooKeeper: Client
> > > > > > >> > environment:java.io.tmpdir=3D/tmp
> > > > > > >> > 10/11/21 02:49:24 INFO zookeeper.ZooKeeper: Client
> > > > > > >> > environment:java.compiler=3D<NA>
> > > > > > >> >
> > > > > > >> > As you can see=2C /opt/hbase/hbase-0.20.6.jar is included =
in the
> > > > > > classpath.
> > > > > > >> > What else could be it?
> > > > > > >> >
> > > > > > >> > thanks=2C
> > > > > > >> > hari
> > > > > > >>
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
 		 	   		  =

--_62ebe1d2-c04a-47f7-b87c-229c7f6b3553_--