giraph-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From KAUSHIK SARKAR <countkaus...@gmail.com>
Subject Re: SimpleShortestPathsVertex does not work when compiled separately
Date Thu, 02 Aug 2012 03:20:43 GMT
Hi,

I found an alternative programmatic solution to the problem.
We need to modify the run() method in the following way -

...
@Override
    public int run(String[] argArray) throws Exception {
        Preconditions.checkArgument(argArray.length == 4,
            "run: Must have 4 arguments <input path> <output path> " +
            "<source vertex id> <# of workers>");

        GiraphJob job = new GiraphJob(getConf(), getClass().getName());
        // This is the addition - it will make hadoop look for other
classes in the same jar that contains this class
        *job.getInternalJob().setJarByClass(getClass());*
        job.setVertexClass(getClass());
        ...
   }

After this hadoop will look for missing classes in the same jar that
contains the class returned by getClass(), and we will not need to add the
job jar name separately to the -libjars option (as described by Marcin's
stackoverflow post).

I think Giraph may add a constructor like -

GiraphJob(Configuration conf, Class jobClass, String jobName) {
      this(conf, jobName);
      this.getInternalJob().setJarByClass(jobClass);
}

in tradition of the Hadoop constructor -

public JobConf(Configuration conf, Class exampleClass) {
    this(conf);
    setJarByClass(exampleClass);
  }

It will solve the rather annoying class-not-found problem.

Regards,
Kaushik

On Tue, Jul 24, 2012 at 11:31 AM, KAUSHIK SARKAR <countkaushik@gmail.com>wrote:

> Hi,
>
> Marcin's solution worked for me. Thanks!
>
> I didn't try out addFileToClassPath solution, but I think it should work.
> Thanks for the suggestion!
>
> As Marcin has already suggested, it is not very much clear why we have to
> do it like this and why the normal call to hadoop jar doesn't work. I guess
> it has more to do with the way Java (or Hadoop) loads class files than
> Giraph. However, I am a total novice in Java and would like to know the
> reason.
>
> N.B. I tried with the latest development snapshot of the Giraph
> (giraph-0.2) , and normal call to hadoop jar  didn't work with it as well
> (of course Marcin's solution worked).
>
> Regards,
> Kaushik
>
> On Tue, Jul 24, 2012 at 6:22 AM, Etienne Dumoulin <
> etienne.dumoulin@idiro.com> wrote:
>
>> Hi,
>>
>> I cannot explain it.
>> But if it any use, for making it working without specifying the library
>> in command line you can use the distributed cache:
>> DistributedCache#addFileToClassPath
>>
>> Regards,
>>
>> √Čtienne
>>
>>
>> On 24 July 2012 13:37, Marcin Biczak <marcinbiczak@gmail.com> wrote:
>>
>>> Hi
>>>
>>> I had similar problem, here is "my solution",
>>> http://stackoverflow.com/questions/10700853/giraph-shortest-paths-example-classnotfoundexception.
>>> But I still don't know why I have do it like this, maybe someone here can
>>> explain this?
>>>
>>> regards
>>> marcin biczak
>>>
>>>
>>> 2012/7/24 KAUSHIK SARKAR <countkaushik@gmail.com>
>>>
>>>> Hi,
>>>>
>>>> I created a new test project and copied the code from
>>>> SimpleShortestPathsVertex.java file (org.apache.giraph.examples) into it.
>>>> But when I run it, it gives me java.lang.ClassNotFoundException. However,
>>>> if I run the same program from the  giraph-0.1-jar-with-dependencies.jar,
>>>> it runs just fine. Following is the relevant information :
>>>>
>>>> Hadoop version: hadoop-1.0.3 (running in pseudo distributed mode)
>>>> giraph version: giraph-0.1
>>>> jar created with Eclipse 3.3.1
>>>> Output:
>>>>
>>>> kaushik@kaushik-laptop:~/hadoop/hadoop-1.0.3$ bin/hadoop jar
>>>> ../jars/test.jar /shortestPathsInputGraph /shortestPathsOutputGraph 0 3
>>>> 12/07/23 18:29:10 INFO mapred.JobClient: Running job:
>>>> job_201207231222_0021
>>>> 12/07/23 18:29:11 INFO mapred.JobClient:  map 0% reduce 0%
>>>> 12/07/23 18:29:20 INFO mapred.JobClient: Task Id :
>>>> attempt_201207231222_0021_m_000005_0, Status : FAILED
>>>> java.lang.RuntimeException: java.lang.RuntimeException:
>>>> java.lang.ClassNotFoundException:
>>>> test.giraph.SimpleShortestPathsVertex$SimpleShortestPathsVertexOutputFormat
>>>>         at
>>>> org.apache.hadoop.conf.Configuration.getClass(Configuration.java:899)
>>>>         at
>>>> org.apache.giraph.graph.BspUtils.getVertexOutputFormatClass(BspUtils.java:134)
>>>>         at
>>>> org.apache.giraph.bsp.BspOutputFormat.getOutputCommitter(BspOutputFormat.java:56)
>>>>         at org.apache.hadoop.mapred.Task.initialize(Task.java:515)
>>>>         at org.apache.hadoop.mapred.MapTask.run(MapTask.java:353)
>>>>         at org.apache.hadoop.mapred.Child$4.run(Child.java:255)
>>>>         at java.security.AccessController.doPrivileged(Native Method)
>>>>         at javax.security.auth.Subject.doAs(Subject.java:396)
>>>>         at
>>>> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1121)
>>>>         at org.apache.hadoop.mapred.Child.main(Child.java:249)
>>>> Caused by: java.lang.RuntimeException:
>>>> java.lang.ClassNotFoundException:
>>>> test.giraph.SimpleShortestPathsVertex$SimpleShortestPathsVertexOutputFormat
>>>>         at
>>>> org.apache.hadoop.conf.Configuration.getClass(Configuration.java:867)
>>>>         at
>>>> org.apache.hadoop.conf.Configuration.getClass(Configuration.java:891)
>>>>         ... 9 more
>>>> Caused by: java.lang.ClassNotFoundException:
>>>> test.giraph.SimpleShortestPathsVertex$SimpleShortestPathsVertexOutputFormat
>>>>         at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
>>>>         at java.security.AccessController.doPrivileged(Native Method)
>>>>         at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
>>>>         at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
>>>>         at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
>>>>     at java.lang.ClassLoader.loadClass(ClassLoader.java:247)
>>>>     at java.lang.Class.forName0(Native Method)
>>>>     at java.lang.Class.forName(Class.java:247)
>>>>     at
>>>> org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:820)
>>>>     at
>>>> org.apache.hadoop.conf.Configuration.getClass(Configuration.java:865)
>>>>     ... 10 more
>>>>
>>>>
>>>>
>>>> I have checked inside the test/giraph folder of the test.jar file. It
>>>> contains the class
>>>> SimpleShortestPathsVertex$SimpleShortestPathsVertexOutputFormat.
>>>> Any help will be appreciated.
>>>>
>>>> Regards,
>>>> Kaushik
>>>>
>>>
>>>
>>>
>>> --
>>> martin
>>>
>>
>>
>

Mime
View raw message