hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Mikkel Kamstrup Erlandsen <...@statsbiblioteket.dk>
Subject Re: Custom Writables in Hadoop RPC
Date Thu, 24 Aug 2006 13:49:45 GMT
On Fri, 2006-08-18 at 15:03 -0700, Benjamin Reed wrote: 
> Are you constructing the JobConf with a Class?

Yeah, I subclass JobConf, and instantiate that subclass in my jars main.

>  And is your class file in
> a directory on the classpath (rather than the Jar file)? If it is, your
> jar file will NOT be submitted with the job. (I opened a bug on this a
> while ago.)

The jar file which also contains my JobConf subclass was not in the
class path. However I symlinked it in hadoop/lib/ and I was able to get
a little further.

> The easiest way to make sure your jar gets sent is to use the setJar()
> method on JobConf.

I use setJar to point the job at the absolute path of the jar.

More or less I have this working now. My problems are in another area
now :-)



> 
> 
> Mikkel Kamstrup Erlandsen wrote:
> > Im using a custom Writable in a proxy server feeding records to
> > hadoop.mapred.RecordReaders. The proxy talks hadoop rpc with the record
> > reader.
> >
> > This works fine when I run with mapred.job.tracker="local", but I get a
> > ClassNotFoundException when i run distributed:
> >
> > 282106-2006-08-17 15:02:34,609 INFO org.apache.hadoop.ipc.Client: Client
> > connection to 127.0.0.1:19093 caught: java.lang.RuntimeException:
> > readObject can't find class
> > 282266-java.lang.RuntimeException: readObject can't find class
> > 282322- at
> > org.apache.hadoop.io.ObjectWritable.readObject(ObjectWritable.java:178)
> > 282398- at
> > org.apache.hadoop.io.ObjectWritable.readFields(ObjectWritable.java:60)
> > 282473- at org.apache.hadoop.ipc.Client$Connection.run(Client.java:170)
> > 282538:Caused by: java.lang.ClassNotFoundException:
> > dk.statsbiblioteket.sbandex.mapred.WritableRecord
> > 282633- at java.net.URLClassLoader$1.run(URLClassLoader.java:200)
> > 282692- at java.security.AccessController.doPrivileged(Native Method)
> > 282755- at java.net.URLClassLoader.findClass(URLClassLoader.java:188)
> > 282818- at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
> > 282876- at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:268)
> > 282942- at java.lang.ClassLoader.loadClass(ClassLoader.java:251)
> > 283000- at java.lang.ClassLoader.loadClassInternal(ClassLoader.java:319)
> > 283066- at java.lang.Class.forName0(Native Method)
> > 283110- at java.lang.Class.forName(Class.java:242)
> > 283154- at
> > org.apache.hadoop.io.ObjectWritable.readObject(ObjectWritable.java:174)
> > 283230- ... 2 more
> >
> >
> > Do I have to add the module containing
> > dk.statsbiblioteket.sbandex.mapred.WritableRecord to Hadoops classpath?
> > I would assume that "hadoop jar myPackage.jar" would import everything
> > from myPackage.jar...
> >
> > Cheers,
> > Mikkel
> >
> >   
> 


Mime
View raw message