Yes it actually is in the JAR file:

mbp-rvesse:target rvesse$ jar -tf /Users/rvesse/Documents/Work/Code/giraph/giraph-examples/target/giraph-examples-1.1.0-SNAPSHOT-for-hadoop-2.2.0-jar-with-dependencies.jar | grep org.apache.giraph.yarn.GiraphApplicationMaster
org/apache/giraph/yarn/GiraphApplicationMaster$1.class
org/apache/giraph/yarn/GiraphApplicationMaster$LaunchContainerRunnable.class
org/apache/giraph/yarn/GiraphApplicationMaster$NMCallbackHandler.class
org/apache/giraph/yarn/GiraphApplicationMaster$RMCallbackHandler.class
org/apache/giraph/yarn/GiraphApplicationMaster.class

Rob

From: Rafal Wojdyla <ravwojdyla@gmail.com>
Reply-To: <user@giraph.apache.org>
Date: Monday, 9 December 2013 10:24
To: <user@giraph.apache.org>
Subject: Re: GiraphApplicationMaster not found (WAS: Giraph and Hadoop 2.2.0)

Hi all!

I had the same problems as you guys:

@Rob:
Did you check if org.apache.giraph.yarn.GiraphApplicationMaster is actually in the jar file - it wasn't it my case - I'm using custom profile in maven.
Do: jar -tf /Users/rvesse/Documents/Work/Code/giraph/giraph-examples/target/giraph-examples-1.1.0-SNAPSHOT-for-hadoop-2.2.0-jar-with-dependencies.jar | grep org.apache.giraph.yarn.GiraphApplicationMaster

@Vishal:
I got this error few times when cluster was highly utilized, problem disappeared when I used queue with free resources - this may be a problem but I haven't investigated it.

Unfortunately I wasn't able to successfully run Giraph application anyway, due to: https://issues.apache.org/jira/browse/GIRAPH-806

Cheers - Rafal


On Sun, Dec 8, 2013 at 7:23 PM, Eli Reisman <apache.mailbox@gmail.com> wrote:
Hi all,

It looks to me like you want to run the giraph-core jar at the beginning of the "hadoop jar" call and supply the giraph-examples jar with -yj argument.

Also, you can place any -D type configuration settings you're sick of supplying at the command line into giraph-site.xml just as you would with Hadoop settings in one of its own site.xml files, and place these where your local Hadoop client configs are and they should be picked up in your application.

Hope that helps,

Eli



On Thu, Nov 28, 2013 at 12:21 PM, Vishal Mishra <mishravishal@gmail.com> wrote:
I had a mistake in my command (moronic copy/paste zookeeper settings error).
Updated and now hitting the following:

....
13/11/28 12:02:14 INFO yarn.GiraphYarnClient: Running Client
13/11/28 12:02:14 INFO client.RMProxy: Connecting to ResourceManager at master/192.168.1.100:8045
13/11/28 12:02:15 INFO yarn.GiraphYarnClient: Got node report from ASM for, nodeId=kreator:46477, nodeAddresskreator:8042, nodeRackName/default-rack, nodeNumContainers7
13/11/28 12:02:15 INFO yarn.GiraphYarnClient: Got node report from ASM for, nodeId=exotica:46645, nodeAddressexotica:8042, nodeRackName/default-rack, nodeNumContainers8
Exception in thread "main" java.lang.RuntimeException: Giraph job requires 2 containers to run; cluster only hosts 15
at org.apache.giraph.yarn.GiraphYarnClient.checkPerNodeResourcesAvailable(GiraphYarnClient.java:230)
at org.apache.giraph.yarn.GiraphYarnClient.run(GiraphYarnClient.java:125)
....

It seems like some config is missing from our hadoop setup. 
Also, note that I'm working from the trunk branch.

Thanx,
Vishal


On Wed, Nov 27, 2013 at 11:34 AM, Vishal Mishra <mishravishal@gmail.com> wrote:
Hey Guys,
I'm trying giraph 1.0.0 on hadoop 2.2.0 (2-node cluster) but hitting the same problem as outlined in this thread. Was there a resolution? If yes, can you please share?
Completely new to giraph and hadoop so please excuse any omissions. Our cluster is fully functional and running mapreduce and hive jobs cleanly.

Command:
hadoop jar $GIRAPH_HOME/giraph-examples/target/giraph-examples-1.1.0-SNAPSHOT-for-hadoop-2.2.0-jar-with-dependencies.jar org.apache.giraph.GiraphRunner -D giraph.zkList="zkNode.net:2081" org.apache.giraph.examples.SimpleShortestPathsComputation -vif org.apache.giraph.io.formats.JsonLongDoubleFloatDoubleVertexInputFormat -vip /user/hduser/rrdata/tiny_graph.txt -vof org.apache.giraph.io.formats.IdWithValueTextOutputFormat -op /user/hduser/rrdata/output -w 1 -yj $GIRAPH_HOME/giraph-examples/target/giraph-examples-1.1.0-SNAPSHOT-for-hadoop-2.2.0-jar-with-dependencies.jar

Trimmed Console Output:
13/11/27 11:19:58 INFO yarn.GiraphYarnClient: appattempt_1385518875931_0002_000001, State: ACCEPTED, Containers used: 1
13/11/27 11:20:00 ERROR yarn.GiraphYarnClient: Giraph: org.apache.giraph.examples.SimpleShortestPathsComputation reports FAILED state, diagnostics show: Application application_1385518875931_0002 failed 2 times due to AM Container for appattempt_1385518875931_0002_000002 exited with  exitCode: 1 due to: Exception from container-launch: 
org.apache.hadoop.util.Shell$ExitCodeException: 
at org.apache.hadoop.util.Shell.runCommand(Shell.java:464)
at org.apache.hadoop.util.Shell.run(Shell.java:379)
at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:589)
at org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:195)
at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:283)
at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:79)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:744)


Thanx,
Vishal


On Tue, Nov 19, 2013 at 3:16 PM, Mohammad Islam <mislam77@yahoo.com> wrote:
Rob,
I understand your frustration.
I'm pre-occupied with some other stuffs in next couple of days.
If no one replied by then, I will recreate the situation here and try to give you more concrete help.

Regards,
Mohammad


On Tuesday, November 19, 2013 2:56 AM, Rob Vesse <rvesse@dotnetrdf.org> wrote:
Mohammad

Comments inline:

From: Mohammad Islam <mislam77@yahoo.com>
Reply-To: <user@giraph.apache.org>
Date: Monday, 18 November 2013 22:24
To: "user@giraph.apache.org" <user@giraph.apache.org>
Subject: Re: GiraphApplicationMaster not found (WAS: Giraph and Hadoop 2.2.0)

I used the following shell script to run PR where my current working directory contains the graph*.jar in Hadoop 2.1.1.
. You can try something similar.

I already am.




HAD_VERSION=2.1.1-SNAPSHOT
hadoop fs -rmr out
export GIRAPH_HOME=/home/mislam/softwares/giraph
#export GIRAPH_CORE=$GIRAPH_HOME/giraph-core/target/giraph-1.1.0-SNAPSHOT-for-hadoop-$HAD_VERSION-jar-with-dependencies.jar
#export GIRAPH_EXAMPLE=$GIRAPH_HOME/giraph-examples/target/giraph-examples-1.1.0-SNAPSHOT-for-hadoop-$HAD_VERSION-jar-with-dependencies.jar


CLASSPATH=$CLASSPATH:$GIRAPH_HOME/conf

What is the point of putting the conf directory on the CLASSPATH?  I assume this is only to pick up the giraph configuration but at the same time there is nothing to stop me putting my Giraph specific config in one of the other Hadoop configuration files e.g. core-site.xml?

#CLASSPATH=$CLASSPATH:$GIRAPH_CORE
#CLASSPATH=$CLASSPATH:$GIRAPH_EXAMPLE

export CLASSPATH=$CLASSPATH
export HADOOP_CLASSPATH=$HADOOP_CLASSPATH:$CLASSPATH

echo $HADOOP_CLASSPATH

I assume the point of the above is to ensure that the Giraph config file is on the class path Hadoop actually uses but adding this makes no difference to me, I get exactly the same failure as before.  Modifying the logic to add the examples JAR onto the class path does not help either.



hadoop --config $HADOOP_HOME/etc/hadoop jar giraph-examples-1.1.0-SNAPSHOT-for-hadoop-$HAD_VERSION-jar-with-dependencies.jar  org.apache.giraph.GiraphRunner org.apache.giraph.examples.SimplePageRankComputation -c org.apache.giraph.combiner.DoubleSumCombiner -w 1 -yh 1024 -yj giraph-examples-1.1.0-SNAPSHOT-for-hadoop-$HAD_VERSION-jar-with-dependencies.jar -vif org.apache.giraph.io.formats.JsonLongDoubleFloatDoubleVertexInputFormat -vof org.apache.giraph.io.formats.IdWithValueTextOutputFormat -mc org.apache.giraph.examples.SimplePageRankComputation\$SimplePageRankMasterCompute -vip input -op out

This is not the same command that I am running, also DoubleSumCombiner gives a ClassNotFoundException, you actually want DoubleSumMessageCombiner.  If I adapt my script to run your command I experience the exact same error as I do trying to run the command I showed so I am still nowhere.

I'm getting a little frustrated that no-one has any idea what the problem is here, either this is a simple misconfiguration on my part or a bug in Giraph for Hadoop 2.2.0 ?  Given the thread so far and the fact that I can run MR based Giraph on Hadoop 1.2.1 this is looking increasingly unlikely I'm starting to lean heavily towards the latter option.

Rob



On Monday, November 18, 2013 2:24 AM, Rob Vesse <rvesse@dotnetrdf.org> wrote:
Hey all

Bumping this thread because I am still completely blocked on this and hoping someone can help me.  As explained earlier in the thread I have Giraph built for pure YARN on stable Hadoop 2.2.0 release like so:

mvn –Phadoop_yarn –Dhadoop.version=2.2.0 –DskipTests=true clean install

But when I try and run the simple shortest paths demo like so:

$> bin/hadoop jar /Users/rvesse/Documents/Work/Code/giraph/giraph-examples/target/giraph-examples-1.1.0-SNAPSHOT-for-hadoop-2.2.0-jar-with-dependencies.jar org.apache.giraph.GiraphRunner org.apache.giraph.examples.SimpleShortestPathsComputation -vif org.apache.giraph.io.formats.JsonLongDoubleFloatDoubleVertexInputFormat -vip /user/rvesse/tiny_graph.txt -vof org.apache.giraph.io.formats.IdWithValueTextOutputFormat -op /user/rvesse/shortestpaths -w 1 -yj /Users/rvesse/Documents/Work/Code/giraph/giraph-examples/target/giraph-examples-1.1.0-SNAPSHOT-for-hadoop-2.2.0-jar-with-dependencies.jar

The application fails and the YARN logs show the following error:

Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/giraph/yarn/GiraphApplicationMaster
Caused by: java.lang.ClassNotFoundException: org.apache.giraph.yarn.GiraphApplicationMaster
	at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
	at java.security.AccessController.doPrivileged(Native Method)
	at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
	at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
	at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
	at java.lang.ClassLoader.loadClass(ClassLoader.java:247)

Any ideas why the GiraphApplicationMaster would not be found?  Have I just got something basic wrong in my configuration or is this a bug in Giraph?

I found an earlier thread on the same topic (http://mail-archives.apache.org/mod_mbox/giraph-user/201307.mbox/%3c51E5A4D1.40006@googlemail.com%3e) but it was never resolved hence why I have now changed the topic in the hope it will jog someone's memory

Cheers,

Rob

From: Rob Vesse <rvesse@dotnetrdf.org>
Reply-To: <user@giraph.apache.org>
Date: Friday, 15 November 2013 14:14
To: <user@giraph.apache.org>
Subject: Re: Giraph and Hadoop 2.2.0

Adding the –yj argument to specify the YARN JARs doesn't make any difference.

Here is the complete console output from my run:

bin/hadoop jar /Users/rvesse/Documents/Work/Code/giraph/giraph-examples/target/giraph-examples-1.1.0-SNAPSHOT-for-hadoop-2.2.0-jar-with-dependencies.jar org.apache.giraph.GiraphRunner org.apache.giraph.examples.SimpleShortestPathsComputation -vif org.apache.giraph.io.formats.JsonLongDoubleFloatDoubleVertexInputFormat -vip /user/rvesse/tiny_graph.txt -vof org.apache.giraph.io.formats.IdWithValueTextOutputFormat -op /user/rvesse/shortestpaths/3 -w 1 -yj /Users/rvesse/Documents/Work/Code/giraph/giraph-examples/target/giraph-examples-1.1.0-SNAPSHOT-for-hadoop-2.2.0-jar-with-dependencies.jar
2013-11-15 14:11:22.897 java[4877:1203] Unable to load realm info from SCDynamicStore
13/11/15 14:11:22 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
13/11/15 14:11:24 INFO utils.ConfigurationUtils: No edge input format specified. Ensure your InputFormat does not require one.
13/11/15 14:11:24 INFO utils.ConfigurationUtils: No edge output format specified. Ensure your OutputFormat does not require one.
13/11/15 14:11:24 INFO yarn.GiraphYarnClient: Final output path is: hdfs://localhost/user/rvesse/shortestpaths/3
13/11/15 14:11:24 INFO yarn.GiraphYarnClient: Running Client
13/11/15 14:11:24 INFO client.RMProxy: Connecting to ResourceManager at /0.0.0.0:8032
13/11/15 14:11:24 INFO yarn.GiraphYarnClient: Got node report from ASM for, nodeId=mbp-rvesse.home:49981, nodeAddressmbp-rvesse.home:8042, nodeRackName/default-rack, nodeNumContainers0
13/11/15 14:11:24 INFO yarn.GiraphYarnClient: Defaulting per-task heap size to 1024MB.
13/11/15 14:11:24 INFO yarn.GiraphYarnClient: Obtained new Application ID: application_1384523910015_0004
13/11/15 14:11:24 INFO Configuration.deprecation: mapred.job.id is deprecated. Instead, use mapreduce.job.id
13/11/15 14:11:24 INFO yarn.GiraphYarnClient: Set the environment for the application master
13/11/15 14:11:24 INFO yarn.GiraphYarnClient: Environment for AM :{CLASSPATH=${CLASSPATH}:./*:$HADOOP_CONF_DIR:$HADOOP_COMMON_HOME/share/hadoop/common/*:$HADOOP_COMMON_HOME/share/hadoop/common/lib/*:$HADOOP_HDFS_HOME/share/hadoop/hdfs/*:$HADOOP_HDFS_HOME/share/hadoop/hdfs/lib/*:$HADOOP_YARN_HOME/share/hadoop/yarn/*:$HADOOP_YARN_HOME/share/hadoop/yarn/lib/*:$HADOOP_MAPRED_HOME/share/hadoop/mapreduce/*,$HADOOP_MAPRED_HOME/share/hadoop/mapreduce/lib/*}
13/11/15 14:11:24 INFO yarn.GiraphYarnClient: buildLocalResourceMap ....
13/11/15 14:11:24 INFO Configuration.deprecation: mapred.output.dir is deprecated. Instead, use mapreduce.output.fileoutputformat.outputdir
13/11/15 14:11:25 INFO yarn.YarnUtils: Registered file in LocalResources :: hdfs://localhost/user/rvesse/giraph_yarn_jar_cache/application_1384523910015_0004/giraph-conf.xml
13/11/15 14:11:25 INFO yarn.GiraphYarnClient: LIB JARS :/Users/rvesse/Documents/Work/Code/giraph/giraph-examples/target/giraph-examples-1.1.0-SNAPSHOT-for-hadoop-2.2.0-jar-with-dependencies.jar
13/11/15 14:11:25 INFO yarn.YarnUtils: Class path name .
13/11/15 14:11:25 INFO yarn.YarnUtils: base path checking .
13/11/15 14:11:25 INFO yarn.YarnUtils: Class path name null
13/11/15 14:11:25 INFO yarn.YarnUtils: base path checking null
13/11/15 14:11:25 INFO yarn.YarnUtils: Class path name /Users/rvesse/Documents/Apps/hadoop-2.2.0/etc/hadoop
13/11/15 14:11:25 INFO yarn.YarnUtils: base path checking /Users/rvesse/Documents/Apps/hadoop-2.2.0/etc/hadoop
13/11/15 14:11:25 INFO yarn.YarnUtils: Class path name /Users/rvesse/Documents/Apps/hadoop-2.2.0/share/hadoop/common/lib/*
13/11/15 14:11:25 INFO yarn.YarnUtils: base path checking /Users/rvesse/Documents/Apps/hadoop-2.2.0/share/hadoop/common/lib
13/11/15 14:11:25 INFO yarn.YarnUtils: Class path name /Users/rvesse/Documents/Apps/hadoop-2.2.0/share/hadoop/common/*
13/11/15 14:11:25 INFO yarn.YarnUtils: base path checking /Users/rvesse/Documents/Apps/hadoop-2.2.0/share/hadoop/common
13/11/15 14:11:25 INFO yarn.YarnUtils: Class path name /Users/rvesse/Documents/Apps/hadoop-2.2.0/share/hadoop/hdfs
13/11/15 14:11:25 INFO yarn.YarnUtils: base path checking /Users/rvesse/Documents/Apps/hadoop-2.2.0/share/hadoop/hdfs
13/11/15 14:11:25 INFO yarn.YarnUtils: Class path name /Users/rvesse/Documents/Apps/hadoop-2.2.0/share/hadoop/hdfs/lib/*
13/11/15 14:11:25 INFO yarn.YarnUtils: base path checking /Users/rvesse/Documents/Apps/hadoop-2.2.0/share/hadoop/hdfs/lib
13/11/15 14:11:25 INFO yarn.YarnUtils: Class path name /Users/rvesse/Documents/Apps/hadoop-2.2.0/share/hadoop/hdfs/*
13/11/15 14:11:25 INFO yarn.YarnUtils: base path checking /Users/rvesse/Documents/Apps/hadoop-2.2.0/share/hadoop/hdfs
13/11/15 14:11:25 INFO yarn.YarnUtils: Class path name /Users/rvesse/Documents/Apps/hadoop-2.2.0/share/hadoop/yarn/lib/*
13/11/15 14:11:25 INFO yarn.YarnUtils: base path checking /Users/rvesse/Documents/Apps/hadoop-2.2.0/share/hadoop/yarn/lib
13/11/15 14:11:25 INFO yarn.YarnUtils: Class path name /Users/rvesse/Documents/Apps/hadoop-2.2.0/share/hadoop/yarn/*
13/11/15 14:11:25 INFO yarn.YarnUtils: base path checking /Users/rvesse/Documents/Apps/hadoop-2.2.0/share/hadoop/yarn
13/11/15 14:11:25 INFO yarn.YarnUtils: Class path name /Users/rvesse/Documents/Apps/hadoop-2.2.0/share/hadoop/mapreduce/lib/*
13/11/15 14:11:25 INFO yarn.YarnUtils: base path checking /Users/rvesse/Documents/Apps/hadoop-2.2.0/share/hadoop/mapreduce/lib
13/11/15 14:11:25 INFO yarn.YarnUtils: Class path name /Users/rvesse/Documents/Apps/hadoop-2.2.0/share/hadoop/mapreduce/*
13/11/15 14:11:25 INFO yarn.YarnUtils: base path checking /Users/rvesse/Documents/Apps/hadoop-2.2.0/share/hadoop/mapreduce
13/11/15 14:11:25 INFO yarn.YarnUtils: Class path name /contrib/capacity-scheduler/*.jar
13/11/15 14:11:25 INFO yarn.YarnUtils: base path checking /contrib/capacity-scheduler
13/11/15 14:11:25 INFO yarn.GiraphYarnClient: ApplicationSumbissionContext for GiraphApplicationMaster launch container is populated.
13/11/15 14:11:25 INFO yarn.GiraphYarnClient: Submitting application to ASM
13/11/15 14:11:25 INFO impl.YarnClientImpl: Submitted application application_1384523910015_0004 to ResourceManager at /0.0.0.0:8032
13/11/15 14:11:25 INFO yarn.GiraphYarnClient: Got new appId after submission :application_1384523910015_0004
13/11/15 14:11:25 INFO yarn.GiraphYarnClient: GiraphApplicationMaster container request was submitted to ResourceManager for job: Giraph: org.apache.giraph.examples.SimpleShortestPathsComputation
13/11/15 14:11:26 INFO yarn.GiraphYarnClient: Giraph: org.apache.giraph.examples.SimpleShortestPathsComputation, Elapsed: 0.82 secs
13/11/15 14:11:26 INFO yarn.GiraphYarnClient: appattempt_1384523910015_0004_000001, State: ACCEPTED, Containers used: 1
13/11/15 14:11:29 ERROR yarn.GiraphYarnClient: Giraph: org.apache.giraph.examples.SimpleShortestPathsComputation reports FAILED state, diagnostics show: Application application_1384523910015_0004 failed 2 times due to AM Container for appattempt_1384523910015_0004_000002 exited with  exitCode: 1 due to: Exception from container-launch: 
org.apache.hadoop.util.Shell$ExitCodeException: 
at org.apache.hadoop.util.Shell.runCommand(Shell.java:464)
at org.apache.hadoop.util.Shell.run(Shell.java:379)
at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:589)
at org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:195)
at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:283)
at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:79)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
at java.util.concurrent.FutureTask.run(FutureTask.java:138)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918)
at java.lang.Thread.run(Thread.java:695)


.Failing this attempt.. Failing the application.
13/11/15 14:11:29 INFO yarn.GiraphYarnClient: Cleaning up HDFS distributed cache directory for Giraph job.
13/11/15 14:11:29 INFO yarn.GiraphYarnClient: Completed Giraph: org.apache.giraph.examples.SimpleShortestPathsComputation: FAILED, total running time: 0 minutes, 3 seconds.

And I see the same error in the logs:

Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/giraph/yarn/GiraphApplicationMaster
Caused by: java.lang.ClassNotFoundException: org.apache.giraph.yarn.GiraphApplicationMaster
	at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
	at java.security.AccessController.doPrivileged(Native Method)
	at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
	at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
	at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
	at java.lang.ClassLoader.loadClass(ClassLoader.java:247)

Any more ideas?

Rob

From: Mohammad Islam <mislam77@yahoo.com>
Reply-To: <user@giraph.apache.org>
Date: Thursday, 14 November 2013 20:36
To: "user@giraph.apache.org" <user@giraph.apache.org>
Subject: Re: Giraph and Hadoop 2.2.0

I tried the following command few months back. I found you are not passing some params.

bin/hadoop --config etc/hadoop jar /export/home/clei/giraph/giraph-1.0.0/giraph-examples/target/giraph-examples-1.?.0-for-hadoop-???-jar-with-dependencies.jar org.apache.giraph.GiraphRunner org.apache.giraph.examples.SimplePageRankVertexOpt -c org.apache.giraph.combiner.DoubleSumCombiner -w 35 -yh 20480 -yj giraph-examples-1.?.?-for-hadoop-2.??-jar-with-dependencies.jar -vif org.apache.giraph.io.formats.JsonLongDoubleFloatDoubleVertexInputFormat -of org.apache.giraph.io.formats.IdWithValueTextOutputFormat -vip /user/me/input -op /user/me/output_35w_20G_80m_1


On Thursday, November 14, 2013 3:22 AM, Rob Vesse <rvesse@dotnetrdf.org> wrote:
It's the same command line I showed earlier, just trying to run the example application:

mbp-rvesse:hadoop-2.2.0 rvesse$ bin/hadoop jar /Users/rvesse/Documents/Work/Code/giraph/giraph-examples/target/giraph-examples-1.1.0-SNAPSHOT-for-hadoop-2.2.0-jar-with-dependencies.jar org.apache.giraph.GiraphRunner org.apache.giraph.examples.SimpleShortestPathsComputation -vif org.apache.giraph.io.formats.JsonLongDoubleFloatDoubleVertexInputFormat -vip /user/rvesse/tiny_graph.txt -vof org.apache.giraph.io.formats.IdWithValueTextOutputFormat -op /user/rvesse/shortestpaths/2 -w 1

Rob

From: Mohammad Islam <mislam77@yahoo.com>
Reply-To: <user@giraph.apache.org>
Date: Thursday, 14 November 2013 05:57
To: "user@giraph.apache.org" <user@giraph.apache.org>
Subject: Re: Giraph and Hadoop 2.2.0

What are you running? PR?
What is the command line?



On Wednesday, November 13, 2013 8:30 AM, Rob Vesse <rvesse@dotnetrdf.org> wrote:
So I fixed the previous issue I was having which I reported and submitted a patch as GIRAPH-795

I then went ahead with installing and configuring my external Zookeeper appropriately and tried running again but now the YARN application fails and I find the following error in the logs:

Exception in thread "main" java.lang.NoClassDefFoundError:
 org/apache/giraph/yarn/GiraphApplicationMaster
Caused by: java.lang.ClassNotFoundException: org.apache.giraph.yarn.GiraphApplicationMaster
	at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
	at java.security.AccessController.doPrivileged(Native Method)
	at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
	at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
	at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
	at java.lang.ClassLoader.loadClass(ClassLoader.java:247)

Anybody have idea what the cause of this one might be or how to fix it?

Cheers,

Rob

From: Rob Vesse <rvesse@dotnetrdf.org>
Reply-To: <user@giraph.apache.org>
Date: Wednesday, 13 November 2013 15:35
To: <user@giraph.apache.org>
Subject: Re: Giraph and Hadoop 2.2.0

Mohammad

This got me a build that at least attempts to start to run but not I see the following:

mbp-rvesse:hadoop-2.2.0 rvesse$ bin/hadoop jar /Users/rvesse/Documents/Work/Code/giraph/giraph-examples/target/giraph-examples-1.1.0-SNAPSHOT-for-hadoop-2.2.0-jar-with-dependencies.jar org.apache.giraph.GiraphRunner org.apache.giraph.examples.SimpleShortestPathsComputation -vif org.apache.giraph.io.formats.JsonLongDoubleFloatDoubleVertexInputFormat -vip /user/rvesse/tiny_graph.txt -vof org.apache.giraph.io.formats.IdWithValueTextOutputFormat -op /user/rvesse/shortestpaths/2 -w 1
2013-11-13 15:13:29.908 java[17402:1203] Unable to load realm info from SCDynamicStore
13/11/13 15:13:29 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
13/11/13 15:13:31 INFO utils.ConfigurationUtils: No edge input format specified. Ensure your InputFormat does not require one.
13/11/13 15:13:31 INFO utils.ConfigurationUtils: No edge output format specified. Ensure your OutputFormat does not require one.
13/11/13 15:13:31 INFO yarn.GiraphYarnClient: Final output path is: hdfs://localhost/user/rvesse/shortestpaths/2
Exception in thread "main" java.lang.NullPointerException
at org.apache.giraph.yarn.GiraphYarnClient.checkJobLocalZooKeeperSupported(GiraphYarnClient.java:460)
at org.apache.giraph.yarn.GiraphYarnClient.run(GiraphYarnClient.java:116)
at org.apache.giraph.GiraphRunner.run(GiraphRunner.java:96)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:84)
at org.apache.giraph.GiraphRunner.main(GiraphRunner.java:126)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.util.RunJar.main(RunJar.java:212)

Which indirectly implies that I have some missing configuration to run Giraph on YARN.  I don't think that should be a NPE and it looks to be a trivial fix so I'll try and fix and send a pull request for that.

The error message I should receive if the NPE wasn't happening is the following:

Giraph on YARN does not currentlysupport Giraph-managed ZK instances: use a standalone ZooKeeper.

What configuration options do I need to have set to point Giraph at an external Zookeeper?

Cheers,

Rob

From: Mohammad Islam <mislam77@yahoo.com>
Reply-To: <user@giraph.apache.org>
Date: Wednesday, 13 November 2013 11:15
To: "user@giraph.apache.org" <user@giraph.apache.org>
Subject: Re: Giraph and Hadoop 2.2.0

Hi Rob,
Did you try with this command "mvn -Phadoop_yarn -Dhadoop.version=2.2.0 clean package -DskipTests"?
Regards,
Mohammad


On Wednesday, November 13, 2013 2:55 AM, Rob Vesse <rvesse@dotnetrdf.org> wrote:
Hi all

I'm having trouble getting Giraph to build and run against a Hadoop 2.2.0 installation.  I tried the hadoop_2.0.3 profile with no luck because I get class cast errors when trying to run the examples.

Is there a specific profile I should be building to support Hadoop 2.2.0 or is this not supported yet?

Thanks,

Rob