giraph-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Marcin Biczak <marcinbic...@gmail.com>
Subject java.lang.ClassNotFoundException for VertexOutputFormat
Date Thu, 08 Nov 2012 16:30:06 GMT
Hi

I have very strange problem regarding ClassNotFoundException for vertex
output format (my own implementation). When I run my program I get an
exception:

java.lang.RuntimeException: java.lang.RuntimeException:
java.lang.ClassNotFoundException:
org.test.giraph.utils.writers.generic.SamplingOutputFormat
        at
org.apache.hadoop.conf.Configuration.getClass(Configuration.java:898)
        at
org.apache.giraph.graph.BspUtils.getVertexOutputFormatClass(BspUtils.java:134)
        at
org.apache.giraph.bsp.BspOutputFormat.getOutputCommitter(BspOutputFormat.java:56)
        at org.apache.hadoop.mapred.Task.initialize(Task.java:490)
        at org.apache.hadoop.mapred.MapTask.run(MapTask.java:352)
        at org.apache.hadoop.mapred.Child$4.run(Child.java:259)
        at java.security.AccessController.doPrivileged(Native Method)
        at javax.security.auth.Subject.doAs(Subject.java:396)
        at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1059)
        at org.apache.hadoop.mapred.Child.main(Child.java:253)

I have came across this exception earlier and I've solved it with the use
of "-libjars" parameter in which I pass the path to
giraph-0.2-SNAPSHOT-jar-with-dependencies.jar and myJob.jar (which I
execute). This solution has worked very nice for me, I used it in a number
of programs (with my own InputFormat and OutputFormat), till today. I've
created a small program (TestVertex) to test one solution and I used a
NewOutputFormat created for the test program. When I received
ClassNotFoundException I thought I have made a mistake somewhere, so I've
checked. I couldn't find any bugs, so I replaced the NewOutputFormat with a
one used across other programs (OldOutputFormat). I got the same exception.
for the old one as well. I have cleaned and build the whole myJob.jar and I
was still getting the exception (both for New and Old). So I tested the
original program (same myJob.jar) in which I'm using the OldOutputFormat
and the program works. Since then I tried couple of OutputFormat and none
of them worked (ClassNotFoundException). Then I commented the whole "body"
of TestVertex and I've paste some old working code (with OldOutputFormat).
The only difference between TestVertex and old working code is the class
name. Final I received the ClassNotFoundException again (again the original
code works, I've tested it). It looks like only the TestVertex class has
this problem, no matter what OutputFormat I will use I am always getting
the exception (all other programs which use my OutputFormats work
correctly).

Another interesting aspect is the fact that if I use Giraph provided
OutputFormat (for example JsonBase64VertexOutputFormat) I get a
ClassNotFoundException but for the TestVertex. I've checked the myJob.jar
and all mentioned classes are present in the jar in correct places.

I run out of ideas any suggestions will be greatly appreciated.

Some conf info
I am using an older Giraph revision (1340869) and Hadoop 0.20.203 in
pseudo-distributed mode.
I have created the HADOOP_CLASSPATH which holds path to
giraph-0.2-SNAPSHOT-jar-with-dependencies.jar.
I execute job in the following way: hadoop jar myJob.jar
org.test.giraph.Test -libjars
/path/to/giraph-0.2-SNAPSHOT-jar-with-dependencies.jar,/path/to/myJob.jar
/path/to/input /path/to/output <job params>

regards
marcin

Mime
View raw message