giraph-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Stefan Beskow <Stefan.Bes...@sas.com>
Subject RE: Problem running SimpleShortestPathsComputation on CDH5 with Hadoop 2.2.0
Date Thu, 20 Feb 2014 19:20:51 GMT
Hi Alex.

I tried that, but unfortunately in my case it didn't work so I'm still getting this exception:

Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/giraph/yarn/GiraphYarnTask
Caused by: java.lang.ClassNotFoundException: org.apache.giraph.yarn.GiraphYarnTask
        at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
        at java.security.AccessController.doPrivileged(Native Method)
        at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
        at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:247)
Could not find the main class: org.apache.giraph.yarn.GiraphYarnTask.  Program will exit.

In case I missed something here is what I did:

1. Copied giraph-examples-1.1.0-SNAPSHOT-for-hadoop-2.2.0-cdh5.0.0-beta-2-jar-with-dependencies.jar
to a directory called /users/stbesk/snapshot_from_git/jars
2. Updated hadoop classpath: export HADOOP_CLASSPATH=/users/stbesk/snapshot_from_git/jars
3. Updated command to only use giraph-examples-1.1.0-SNAPSHOT-for-hadoop-2.2.0-cdh5.0.0-beta-2-jar-with-dependencies.jar
since it contains all dependencies:

hadoop jar /users/stbesk/snapshot_from_git/jars/giraph-examples-1.1.0-SNAPSHOT-for-hadoop-2.2.0-cdh5.0.0-beta-2-jar-with-dependencies.jar

org.apache.giraph.GiraphRunner org.apache.giraph.examples.SimpleShortestPathsComputation 
-vif org.apache.giraph.io.formats.JsonLongDoubleFloatDoubleVertexInputFormat 
-vip /user/stbesk/input/tiny-graph.txt 
-vof org.apache.giraph.io.formats.IdWithValueTextOutputFormat 
-op /user/stbesk/output/shortestpathsC2 
-w 2 
-ca giraph.SplitMasterWorker=true 
-ca giraph.zkList=localhost:2181 
-yj giraph-examples-1.1.0-SNAPSHOT-for-hadoop-2.2.0-cdh5.0.0-beta-2-jar-with-dependencies.jar

Did I miss something?

Thanks.
Stefan


-----Original Message-----
From: Alexandre Fonseca [mailto:alexandrejorgefonseca@gmail.com] 
Sent: Thursday, February 20, 2014 1:11 PM
To: user@giraph.apache.org
Subject: Re: Problem running SimpleShortestPathsComputation on CDH5 with Hadoop 2.2.0

The yarn version of Giraph is quite finicky when it comes to detecting the giraph jar. When
it can't find the jar on the new containers you get that exception.

After much experimentation I've had great success with the following (substitute paths and
filenames where needed):

HADOOP_CLASSPATH=/data/b.ajf/giraph hadoop jar /data/b.ajf/giraph/giraph.jar org.apache.giraph.GiraphRunner
org.apache.giraph.examples.SimpleShortestPathsComputation -vip /user/b.ajf/ssp_input -vif
org.apache.giraph.io.formats.JsonLongDoubleFloatDoubleVertexInputFormat -op /user/b.ajf/ssp_output
-vof org.apache.giraph.io.formats.IdWithValueTextOutputFormat -yj giraph.jar

The argument supplied to -yj must be only the name of the jar (no directories). The jar must
be directly accessible from the current classpath or subdirectories (but it doesn't work if
you give the full path of the jar, e.g, HADOOP_CLASSPATH=/data/b.ajf/giraph/giraph.jar).

Also, if you use giraph-examples-with-dependencies you don't need the other one as the examples-with-dependencies
already contain the core.

Cheers,
Alex

On Thursday, February 20, 2014 05:42:45 PM Stefan Beskow wrote:
> Thanks Roman for getting back to me so quickly. I checked with 
> Cloudera and they have a maven repository for CDH5 (version 
> 2.2.0-cdh5.0.0-beta-2) as shown below:
> 
> <repository>
>       <id>cloudera</id>
>       
> <url>https://repository.cloudera.com/artifactory/cloudera-repos/</url>
> </repository>
> 
> I added that to the pom.xml file and ran a new build using the 
> following
> command:
> 
> mvn -Phadoop_yarn -Dhadoop.version=2.2.0-cdh5.0.0-beta-2 clean package 
> -DskipTests
> 
> The program now gets further, but throws the following exception:
> 
> Container: container_1392713839733_0017_01_000002 on
> el01cn04.unx.sas.com_8041
> 
===========================================================================
> ===== LogType: task-2-stderr.log
> LogLength: 637
> Log Contents:
> Exception in thread "main" java.lang.NoClassDefFoundError:
> org/apache/giraph/yarn/GiraphYarnTask Caused by:
> java.lang.ClassNotFoundException: 
> org.apache.giraph.yarn.GiraphYarnTask at
> java.net.URLClassLoader$1.run(URLClassLoader.java:202)
>         at java.security.AccessController.doPrivileged(Native Method)
>         at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
>         at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
>         at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
>         at java.lang.ClassLoader.loadClass(ClassLoader.java:247)
> Could not find the main class: org.apache.giraph.yarn.GiraphYarnTask. 
> Program will exit.
> 
> Do you have any additional suggestions for what I should try?
> 
> Thanks for your help.
> Stefan
> 
> -----Original Message-----
> From: shaposhnik@gmail.com [mailto:shaposhnik@gmail.com] On Behalf Of 
> Roman Shaposhnik Sent: Wednesday, February 19, 2014 4:43 PM
> To: user@giraph.apache.org
> Subject: Re: Problem running SimpleShortestPathsComputation on CDH5 
> with Hadoop 2.2.0 On Wed, Feb 19, 2014 at 1:12 PM, Stefan Beskow 
> <Stefan.Beskow@sas.com>
wrote:
> > Hi.
> > 
> > I checked out and built Giraph for Cloudera CDH5 with Hadoop 2.2.0 
> > using the
> > following:
> > 
> > git clone git://git.apache.org/giraph.git snapshot_from_git cd 
> > snapshot_from_git mvn -Phadoop_yarn -Dhadoop.version=2.2.0 clean 
> > package -DskipTests
> 
> It may help to build against CDH5 directly by:
>    * manually adding repository.cloudera.com to the set of repos
>    * specifying -Dhadoop.version=2.2.0-cdh5.0.0-beta-2
> 
> > When I run the sample application
> > org.apache.giraph.examples.SimpleShortestPathsComputation I get the
> 
> > following exception:
> You need to provide way more logs from the YARN side for us to make 
> sense of it.
> 
> Thanks,
> Roman.



Mime
View raw message