Mailing-List: contact user-help@giraph.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@giraph.apache.org
Received-SPF: pass (nike.apache.org: domain of Stefan.Beskow@sas.com
 designates 216.32.181.186 as permitted sender)
Received-SPF: pass (mail63-ch1: domain of sas.com designates 149.173.6.148 as
 permitted sender) client-ip=149.173.6.148;
 envelope-from=Stefan.Beskow@sas.com;
 helo=mercav06d.na.sas.com ;d.na.sas.com ;
From: Stefan Beskow <Stefan.Beskow@sas.com>
To: "user@giraph.apache.org" <user@giraph.apache.org>
Subject: RE: Problem running SimpleShortestPathsComputation on CDH5 with
 Hadoop 2.2.0
Thread-Topic: Problem running SimpleShortestPathsComputation on CDH5 with
 Hadoop 2.2.0
Thread-Index: Ac8ttz2bT7xNDUaMS0W2ILSgsy2ZwwALituAAAjwMtAAIfWqgAAIcWaA
Date: Thu, 20 Feb 2014 19:20:51 +0000
Message-ID: <41f8cae3775840a3b4cb7375ac1a74e3@MERCMBX25R.na.SAS.com>
References: <50d24338c9164ee2929995e22f532a96@MERCMBX25R.na.SAS.com>
 <CA+ULb+tMdh5yVseqLaQfmvM=_1G=4VayN6ASDypaFMci_zT_aw@mail.gmail.com>
 <32c1ea4f88ec4fd2bc0815b012c0de48@MERCMBX25R.na.SAS.com>
 <2797654.9KU3s8WQsB@pc509946>
In-Reply-To: <2797654.9KU3s8WQsB@pc509946>
Accept-Language: en-US
Content-Language: en-US
Content-Type: text/plain; charset="us-ascii"
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0

Hi Alex.

I tried that, but unfortunately in my case it didn't work so I'm still gett=
ing this exception:

Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/girap=
h/yarn/GiraphYarnTask
Caused by: java.lang.ClassNotFoundException: org.apache.giraph.yarn.GiraphY=
arnTask
        at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
        at java.security.AccessController.doPrivileged(Native Method)
        at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
        at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:247)
Could not find the main class: org.apache.giraph.yarn.GiraphYarnTask.  Prog=
ram will exit.

In case I missed something here is what I did:

1. Copied giraph-examples-1.1.0-SNAPSHOT-for-hadoop-2.2.0-cdh5.0.0-beta-2-j=
ar-with-dependencies.jar to a directory called /users/stbesk/snapshot_from_=
git/jars
2. Updated hadoop classpath: export HADOOP_CLASSPATH=3D/users/stbesk/snapsh=
ot_from_git/jars
3. Updated command to only use giraph-examples-1.1.0-SNAPSHOT-for-hadoop-2.=
2.0-cdh5.0.0-beta-2-jar-with-dependencies.jar since it contains all depende=
ncies:

hadoop jar /users/stbesk/snapshot_from_git/jars/giraph-examples-1.1.0-SNAPS=
HOT-for-hadoop-2.2.0-cdh5.0.0-beta-2-jar-with-dependencies.jar=20
org.apache.giraph.GiraphRunner org.apache.giraph.examples.SimpleShortestPat=
hsComputation=20
-vif org.apache.giraph.io.formats.JsonLongDoubleFloatDoubleVertexInputForma=
t=20
-vip /user/stbesk/input/tiny-graph.txt=20
-vof org.apache.giraph.io.formats.IdWithValueTextOutputFormat=20
-op /user/stbesk/output/shortestpathsC2=20
-w 2=20
-ca giraph.SplitMasterWorker=3Dtrue=20
-ca giraph.zkList=3Dlocalhost:2181=20
-yj giraph-examples-1.1.0-SNAPSHOT-for-hadoop-2.2.0-cdh5.0.0-beta-2-jar-wit=
h-dependencies.jar

Did I miss something?

Thanks.
Stefan


-----Original Message-----
From: Alexandre Fonseca [mailto:alexandrejorgefonseca@gmail.com]=20
Sent: Thursday, February 20, 2014 1:11 PM
To: user@giraph.apache.org
Subject: Re: Problem running SimpleShortestPathsComputation on CDH5 with Ha=
doop 2.2.0

The yarn version of Giraph is quite finicky when it comes to detecting the =
giraph jar. When it can't find the jar on the new containers you get that e=
xception.

After much experimentation I've had great success with the following (subst=
itute paths and filenames where needed):

HADOOP_CLASSPATH=3D/data/b.ajf/giraph hadoop jar /data/b.ajf/giraph/giraph.=
jar org.apache.giraph.GiraphRunner org.apache.giraph.examples.SimpleShortes=
tPathsComputation -vip /user/b.ajf/ssp_input -vif org.apache.giraph.io.form=
ats.JsonLongDoubleFloatDoubleVertexInputFormat -op /user/b.ajf/ssp_output -=
vof org.apache.giraph.io.formats.IdWithValueTextOutputFormat -yj giraph.jar

The argument supplied to -yj must be only the name of the jar (no directori=
es). The jar must be directly accessible from the current classpath or subd=
irectories (but it doesn't work if you give the full path of the jar, e.g, =
HADOOP_CLASSPATH=3D/data/b.ajf/giraph/giraph.jar).

Also, if you use giraph-examples-with-dependencies you don't need the other=
 one as the examples-with-dependencies already contain the core.

Cheers,
Alex

On Thursday, February 20, 2014 05:42:45 PM Stefan Beskow wrote:
> Thanks Roman for getting back to me so quickly. I checked with=20
> Cloudera and they have a maven repository for CDH5 (version=20
> 2.2.0-cdh5.0.0-beta-2) as shown below:
>=20
> <repository>
>       <id>cloudera</id>
>      =20
> <url>https://repository.cloudera.com/artifactory/cloudera-repos/</url>
> </repository>
>=20
> I added that to the pom.xml file and ran a new build using the=20
> following
> command:
>=20
> mvn -Phadoop_yarn -Dhadoop.version=3D2.2.0-cdh5.0.0-beta-2 clean package=
=20
> -DskipTests
>=20
> The program now gets further, but throws the following exception:
>=20
> Container: container_1392713839733_0017_01_000002 on
> el01cn04.unx.sas.com_8041
>=20
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D
> =3D=3D=3D=3D=3D LogType: task-2-stderr.log
> LogLength: 637
> Log Contents:
> Exception in thread "main" java.lang.NoClassDefFoundError:
> org/apache/giraph/yarn/GiraphYarnTask Caused by:
> java.lang.ClassNotFoundException:=20
> org.apache.giraph.yarn.GiraphYarnTask at
> java.net.URLClassLoader$1.run(URLClassLoader.java:202)
>         at java.security.AccessController.doPrivileged(Native Method)
>         at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
>         at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
>         at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
>         at java.lang.ClassLoader.loadClass(ClassLoader.java:247)
> Could not find the main class: org.apache.giraph.yarn.GiraphYarnTask.=20
> Program will exit.
>=20
> Do you have any additional suggestions for what I should try?
>=20
> Thanks for your help.
> Stefan
>=20
> -----Original Message-----
> From: shaposhnik@gmail.com [mailto:shaposhnik@gmail.com] On Behalf Of=20
> Roman Shaposhnik Sent: Wednesday, February 19, 2014 4:43 PM
> To: user@giraph.apache.org
> Subject: Re: Problem running SimpleShortestPathsComputation on CDH5=20
> with Hadoop 2.2.0 On Wed, Feb 19, 2014 at 1:12 PM, Stefan Beskow=20
> <Stefan.Beskow@sas.com>
wrote:
> > Hi.
> >=20
> > I checked out and built Giraph for Cloudera CDH5 with Hadoop 2.2.0=20
> > using the
> > following:
> >=20
> > git clone git://git.apache.org/giraph.git snapshot_from_git cd=20
> > snapshot_from_git mvn -Phadoop_yarn -Dhadoop.version=3D2.2.0 clean=20
> > package -DskipTests
>=20
> It may help to build against CDH5 directly by:
>    * manually adding repository.cloudera.com to the set of repos
>    * specifying -Dhadoop.version=3D2.2.0-cdh5.0.0-beta-2
>=20
> > When I run the sample application
> > org.apache.giraph.examples.SimpleShortestPathsComputation I get the
>=20
> > following exception:
> You need to provide way more logs from the YARN side for us to make=20
> sense of it.
>=20
> Thanks,
> Roman.