giraph-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Vijayakumar Ramdoss <nellaivi...@me.com>
Subject Re: SimpleShortestPathsVertex does not work when compiled separately
Date Sun, 05 Aug 2012 23:25:30 GMT
Thanks Kaushik, it's worked me well.

Sent from my iPad

On Aug 1, 2012, at 11:20 PM, KAUSHIK SARKAR <countkaushik@gmail.com> wrote:

> Hi,
> 
> I found an alternative programmatic solution to the problem.
> We need to modify the run() method in the following way - 
> 
> ...
> @Override
>     public int run(String[] argArray) throws Exception {
>         Preconditions.checkArgument(argArray.length == 4,
>             "run: Must have 4 arguments <input path> <output path> " +
>             "<source vertex id> <# of workers>");
> 
>         GiraphJob job = new GiraphJob(getConf(), getClass().getName());
>         // This is the addition - it will make hadoop look for other classes in the same
jar that contains this class
>         job.getInternalJob().setJarByClass(getClass());
>         job.setVertexClass(getClass());
>         ...
>    }
> 
> After this hadoop will look for missing classes in the same jar that contains the class
returned by getClass(), and we will not need to add the job jar name separately to the -libjars
option (as described by Marcin's stackoverflow post).
> 
> I think Giraph may add a constructor like - 
> 
> GiraphJob(Configuration conf, Class jobClass, String jobName) {
>       this(conf, jobName);
>       this.getInternalJob().setJarByClass(jobClass);
> }
> 
> in tradition of the Hadoop constructor - 
> 
> public JobConf(Configuration conf, Class exampleClass) {
>     this(conf);
>     setJarByClass(exampleClass);
>   }
> 
> It will solve the rather annoying class-not-found problem.
> 
> Regards,
> Kaushik
> 
> On Tue, Jul 24, 2012 at 11:31 AM, KAUSHIK SARKAR <countkaushik@gmail.com> wrote:
> Hi,
> 
> Marcin's solution worked for me. Thanks!
> 
> I didn't try out addFileToClassPath solution, but I think it should work. Thanks for
the suggestion!
> 
> As Marcin has already suggested, it is not very much clear why we have to do it like
this and why the normal call to hadoop jar doesn't work. I guess it has more to do with the
way Java (or Hadoop) loads class files than Giraph. However, I am a total novice in Java and
would like to know the reason.
> 
> N.B. I tried with the latest development snapshot of the Giraph (giraph-0.2) , and normal
call to hadoop jar  didn't work with it as well (of course Marcin's solution worked).
> 
> Regards,
> Kaushik   
> 
> On Tue, Jul 24, 2012 at 6:22 AM, Etienne Dumoulin <etienne.dumoulin@idiro.com>
wrote:
> Hi,
> 
> I cannot explain it. 
> But if it any use, for making it working without specifying the library in command line
you can use the distributed cache:
> DistributedCache#addFileToClassPath
> 
> Regards,
> 
> √Čtienne
> 
> 
> On 24 July 2012 13:37, Marcin Biczak <marcinbiczak@gmail.com> wrote:
> Hi
> 
> I had similar problem, here is "my solution", http://stackoverflow.com/questions/10700853/giraph-shortest-paths-example-classnotfoundexception.
But I still don't know why I have do it like this, maybe someone here can explain this?
> 
> regards
> marcin biczak
> 
> 
> 2012/7/24 KAUSHIK SARKAR <countkaushik@gmail.com>
> Hi,
> 
> I created a new test project and copied the code from SimpleShortestPathsVertex.java
file (org.apache.giraph.examples) into it. But when I run it, it gives me java.lang.ClassNotFoundException.
However, if I run the same program from the  giraph-0.1-jar-with-dependencies.jar, it runs
just fine. Following is the relevant information :
> 
> Hadoop version: hadoop-1.0.3 (running in pseudo distributed mode)
> giraph version: giraph-0.1
> jar created with Eclipse 3.3.1 
> Output:
> 
> kaushik@kaushik-laptop:~/hadoop/hadoop-1.0.3$ bin/hadoop jar ../jars/test.jar /shortestPathsInputGraph
/shortestPathsOutputGraph 0 3
> 12/07/23 18:29:10 INFO mapred.JobClient: Running job: job_201207231222_0021
> 12/07/23 18:29:11 INFO mapred.JobClient:  map 0% reduce 0%
> 12/07/23 18:29:20 INFO mapred.JobClient: Task Id : attempt_201207231222_0021_m_000005_0,
Status : FAILED
> java.lang.RuntimeException: java.lang.RuntimeException: java.lang.ClassNotFoundException:
test.giraph.SimpleShortestPathsVertex$SimpleShortestPathsVertexOutputFormat
>         at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:899)
>         at org.apache.giraph.graph.BspUtils.getVertexOutputFormatClass(BspUtils.java:134)
>         at org.apache.giraph.bsp.BspOutputFormat.getOutputCommitter(BspOutputFormat.java:56)
>         at org.apache.hadoop.mapred.Task.initialize(Task.java:515)
>         at org.apache.hadoop.mapred.MapTask.run(MapTask.java:353)
>         at org.apache.hadoop.mapred.Child$4.run(Child.java:255)
>         at java.security.AccessController.doPrivileged(Native Method)
>         at javax.security.auth.Subject.doAs(Subject.java:396)
>         at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1121)
>         at org.apache.hadoop.mapred.Child.main(Child.java:249)
> Caused by: java.lang.RuntimeException: java.lang.ClassNotFoundException: test.giraph.SimpleShortestPathsVertex$SimpleShortestPathsVertexOutputFormat
>         at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:867)
>         at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:891)
>         ... 9 more
> Caused by: java.lang.ClassNotFoundException: test.giraph.SimpleShortestPathsVertex$SimpleShortestPathsVertexOutputFormat
>         at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
>         at java.security.AccessController.doPrivileged(Native Method)
>         at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
>         at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
>         at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
>     at java.lang.ClassLoader.loadClass(ClassLoader.java:247)
>     at java.lang.Class.forName0(Native Method)
>     at java.lang.Class.forName(Class.java:247)
>     at org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:820)
>     at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:865)
>     ... 10 more
> 
> 
> 
> I have checked inside the test/giraph folder of the test.jar file. It contains the class
SimpleShortestPathsVertex$SimpleShortestPathsVertexOutputFormat.
> Any help will be appreciated.
> 
> Regards,
> Kaushik
> 
> 
> 
> -- 
> martin
> 
> 
> 

Mime
View raw message