giraph-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Vishal Mishra <mishravis...@gmail.com>
Subject Re: GiraphApplicationMaster not found (WAS: Giraph and Hadoop 2.2.0)
Date Thu, 28 Nov 2013 20:21:25 GMT
I had a mistake in my command (moronic copy/paste zookeeper settings error).
Updated and now hitting the following:

....
13/11/28 12:02:14 INFO yarn.GiraphYarnClient: Running Client
13/11/28 12:02:14 INFO client.RMProxy: Connecting to ResourceManager at
master/192.168.1.100:8045
13/11/28 12:02:15 INFO yarn.GiraphYarnClient: Got node report from ASM for,
nodeId=kreator:46477, nodeAddresskreator:8042, nodeRackName/default-rack,
nodeNumContainers7
13/11/28 12:02:15 INFO yarn.GiraphYarnClient: Got node report from ASM for,
nodeId=exotica:46645, nodeAddressexotica:8042, nodeRackName/default-rack,
nodeNumContainers8
Exception in thread "main" java.lang.RuntimeException: Giraph job requires
2 containers to run; cluster only hosts 15
at
org.apache.giraph.yarn.GiraphYarnClient.checkPerNodeResourcesAvailable(GiraphYarnClient.java:230)
at org.apache.giraph.yarn.GiraphYarnClient.run(GiraphYarnClient.java:125)
....

It seems like some config is missing from our hadoop setup.
Also, note that I'm working from the trunk branch.

Thanx,
Vishal


On Wed, Nov 27, 2013 at 11:34 AM, Vishal Mishra <mishravishal@gmail.com>wrote:

> Hey Guys,
> I'm trying giraph 1.0.0 on hadoop 2.2.0 (2-node cluster) but hitting the
> same problem as outlined in this thread. Was there a resolution? If yes,
> can you please share?
> Completely new to giraph and hadoop so please excuse any omissions. Our
> cluster is fully functional and running mapreduce and hive jobs cleanly.
>
> Command:
> hadoop jar
> $GIRAPH_HOME/giraph-examples/target/giraph-examples-1.1.0-SNAPSHOT-for-hadoop-2.2.0-jar-with-dependencies.jar
> org.apache.giraph.GiraphRunner -D giraph.zkList="zkNode.net:2081"
> org.apache.giraph.examples.SimpleShortestPathsComputation -vif
> org.apache.giraph.io.formats.JsonLongDoubleFloatDoubleVertexInputFormat
> -vip /user/hduser/rrdata/tiny_graph.txt -vof
> org.apache.giraph.io.formats.IdWithValueTextOutputFormat -op
> /user/hduser/rrdata/output -w 1 -yj
> $GIRAPH_HOME/giraph-examples/target/giraph-examples-1.1.0-SNAPSHOT-for-hadoop-2.2.0-jar-with-dependencies.jar
>
> Trimmed Console Output:
> 13/11/27 11:19:58 INFO yarn.GiraphYarnClient:
> appattempt_1385518875931_0002_000001, State: ACCEPTED, Containers used: 1
> 13/11/27 11:20:00 ERROR yarn.GiraphYarnClient: Giraph:
> org.apache.giraph.examples.SimpleShortestPathsComputation reports FAILED
> state, diagnostics show: Application application_1385518875931_0002 failed
> 2 times due to AM Container for appattempt_1385518875931_0002_000002 exited
> with  exitCode: 1 due to: Exception from container-launch:
> org.apache.hadoop.util.Shell$ExitCodeException:
> at org.apache.hadoop.util.Shell.runCommand(Shell.java:464)
> at org.apache.hadoop.util.Shell.run(Shell.java:379)
>  at
> org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:589)
> at
> org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:195)
>  at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:283)
> at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:79)
>  at java.util.concurrent.FutureTask.run(FutureTask.java:262)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>  at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:744)
>
>
> Thanx,
> Vishal
>
>
> On Tue, Nov 19, 2013 at 3:16 PM, Mohammad Islam <mislam77@yahoo.com>wrote:
>
>> Rob,
>> I understand your frustration.
>> I'm pre-occupied with some other stuffs in next couple of days.
>> If no one replied by then, I will recreate the situation here and try to
>> give you more concrete help.
>>
>> Regards,
>> Mohammad
>>
>>
>>   On Tuesday, November 19, 2013 2:56 AM, Rob Vesse <rvesse@dotnetrdf.org>
>> wrote:
>>  Mohammad
>>
>> Comments inline:
>>
>> From: Mohammad Islam <mislam77@yahoo.com>
>> Reply-To: <user@giraph.apache.org>
>> Date: Monday, 18 November 2013 22:24
>> To: "user@giraph.apache.org" <user@giraph.apache.org>
>> Subject: Re: GiraphApplicationMaster not found (WAS: Giraph and Hadoop
>> 2.2.0)
>>
>> I used the following shell script to run PR where my current working
>> directory contains the graph*.jar in Hadoop 2.1.1.
>> . You can try something similar.
>>
>>
>> I already am.
>>
>>
>>
>>
>> HAD_VERSION=2.1.1-SNAPSHOT
>> hadoop fs -rmr out
>> export GIRAPH_HOME=/home/mislam/softwares/giraph
>> #export
>> GIRAPH_CORE=$GIRAPH_HOME/giraph-core/target/giraph-1.1.0-SNAPSHOT-for-hadoop-$HAD_VERSION-jar-with-dependencies.jar
>> #export
>> GIRAPH_EXAMPLE=$GIRAPH_HOME/giraph-examples/target/giraph-examples-1.1.0-SNAPSHOT-for-hadoop-$HAD_VERSION-jar-with-dependencies.jar
>>
>>
>> CLASSPATH=$CLASSPATH:$GIRAPH_HOME/conf
>>
>>
>> What is the point of putting the conf directory on the CLASSPATH?  I
>> assume this is only to pick up the giraph configuration but at the same
>> time there is nothing to stop me putting my Giraph specific config in one
>> of the other Hadoop configuration files e.g. core-site.xml?
>>
>> #CLASSPATH=$CLASSPATH:$GIRAPH_CORE
>> #CLASSPATH=$CLASSPATH:$GIRAPH_EXAMPLE
>>
>> export CLASSPATH=$CLASSPATH
>> export HADOOP_CLASSPATH=$HADOOP_CLASSPATH:$CLASSPATH
>>
>> echo $HADOOP_CLASSPATH
>>
>>
>> I assume the point of the above is to ensure that the Giraph config file
>> is on the class path Hadoop actually uses but adding this makes no
>> difference to me, I get exactly the same failure as before.  Modifying the
>> logic to add the examples JAR onto the class path does not help either.
>>
>>
>>
>> hadoop --config $HADOOP_HOME/etc/hadoop jar
>> giraph-examples-1.1.0-SNAPSHOT-for-hadoop-$HAD_VERSION-jar-with-dependencies.jar
>>  org.apache.giraph.GiraphRunner
>> org.apache.giraph.examples.SimplePageRankComputation -c
>> org.apache.giraph.combiner.DoubleSumCombiner -w 1 -yh 1024 -yj
>> giraph-examples-1.1.0-SNAPSHOT-for-hadoop-$HAD_VERSION-jar-with-dependencies.jar
>> -vif
>> org.apache.giraph.io.formats.JsonLongDoubleFloatDoubleVertexInputFormat
>> -vof org.apache.giraph.io.formats.IdWithValueTextOutputFormat -mc
>> org.apache.giraph.examples.SimplePageRankComputation\$SimplePageRankMasterCompute
>> -vip input -op out
>>
>>
>> This is not the same command that I am running, also DoubleSumCombiner
>> gives a ClassNotFoundException, you actually want DoubleSumMessageCombiner.
>>  If I adapt my script to run your command I experience the exact same error
>> as I do trying to run the command I showed so I am still nowhere.
>>
>> I'm getting a little frustrated that no-one has any idea what the problem
>> is here, either this is a simple misconfiguration on my part or a bug in
>> Giraph for Hadoop 2.2.0 ?  Given the thread so far and the fact that I can
>> run MR based Giraph on Hadoop 1.2.1 this is looking increasingly unlikely
>> I'm starting to lean heavily towards the latter option.
>>
>> Rob
>>
>>
>>
>>   On Monday, November 18, 2013 2:24 AM, Rob Vesse <rvesse@dotnetrdf.org>
>> wrote:
>>  Hey all
>>
>> Bumping this thread because I am still completely blocked on this and
>> hoping someone can help me.  As explained earlier in the thread I have
>> Giraph built for pure YARN on stable Hadoop 2.2.0 release like so:
>>
>> mvn –Phadoop_yarn –Dhadoop.version=2.2.0 –DskipTests=true clean install
>>
>> But when I try and run the simple shortest paths demo like so:
>>
>> $> bin/hadoop jar
>> /Users/rvesse/Documents/Work/Code/giraph/giraph-examples/target/giraph-examples-1.1.0-SNAPSHOT-for-hadoop-2.2.0-jar-with-dependencies.jar
>> org.apache.giraph.GiraphRunner
>> org.apache.giraph.examples.SimpleShortestPathsComputation -vif
>> org.apache.giraph.io.formats.JsonLongDoubleFloatDoubleVertexInputFormat
>> -vip /user/rvesse/tiny_graph.txt -vof
>> org.apache.giraph.io.formats.IdWithValueTextOutputFormat -op
>> /user/rvesse/shortestpaths -w 1 -yj
>> /Users/rvesse/Documents/Work/Code/giraph/giraph-examples/target/giraph-examples-1.1.0-SNAPSHOT-for-hadoop-2.2.0-jar-with-dependencies.jar
>>
>> The application fails and the YARN logs show the following error:
>>
>> Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/giraph/yarn/GiraphApplicationMaster
>> Caused by: java.lang.ClassNotFoundException: org.apache.giraph.yarn.GiraphApplicationMaster
>> 	at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
>> 	at java.security.AccessController.doPrivileged(Native Method)
>> 	at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
>> 	at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
>> 	at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
>> 	at java.lang.ClassLoader.loadClass(ClassLoader.java:247)
>>
>>
>> Any ideas why the GiraphApplicationMaster would not be found?  Have I just got something
basic wrong in my configuration or is this a bug in Giraph?
>>
>>
>> I found an earlier thread on the same topic (http://mail-archives.apache.org/mod_mbox/giraph-user/201307.mbox/%3c51E5A4D1.40006@googlemail.com%3e)
but it was never resolved hence why I have now changed the topic in the hope it will jog someone's
memory
>>
>>
>> Cheers,
>>
>>
>> Rob
>>
>>
>> From: Rob Vesse <rvesse@dotnetrdf.org>
>> Reply-To: <user@giraph.apache.org>
>> Date: Friday, 15 November 2013 14:14
>> To: <user@giraph.apache.org>
>> Subject: Re: Giraph and Hadoop 2.2.0
>>
>> Adding the –yj argument to specify the YARN JARs doesn't make any
>> difference.
>>
>> Here is the complete console output from my run:
>>
>> bin/hadoop jar
>> /Users/rvesse/Documents/Work/Code/giraph/giraph-examples/target/giraph-examples-1.1.0-SNAPSHOT-for-hadoop-2.2.0-jar-with-dependencies.jar
>> org.apache.giraph.GiraphRunner
>> org.apache.giraph.examples.SimpleShortestPathsComputation -vif
>> org.apache.giraph.io.formats.JsonLongDoubleFloatDoubleVertexInputFormat
>> -vip /user/rvesse/tiny_graph.txt -vof
>> org.apache.giraph.io.formats.IdWithValueTextOutputFormat -op
>> /user/rvesse/shortestpaths/3 -w 1 -yj
>> /Users/rvesse/Documents/Work/Code/giraph/giraph-examples/target/giraph-examples-1.1.0-SNAPSHOT-for-hadoop-2.2.0-jar-with-dependencies.jar
>> 2013-11-15 14:11:22.897 java[4877:1203] Unable to load realm info from
>> SCDynamicStore
>> 13/11/15 14:11:22 WARN util.NativeCodeLoader: Unable to load
>> native-hadoop library for your platform... using builtin-java classes where
>> applicable
>> 13/11/15 14:11:24 INFO utils.ConfigurationUtils: No edge input format
>> specified. Ensure your InputFormat does not require one.
>> 13/11/15 14:11:24 INFO utils.ConfigurationUtils: No edge output format
>> specified. Ensure your OutputFormat does not require one.
>> 13/11/15 14:11:24 INFO yarn.GiraphYarnClient: Final output path is:
>> hdfs://localhost/user/rvesse/shortestpaths/3
>> 13/11/15 14:11:24 INFO yarn.GiraphYarnClient: Running Client
>> 13/11/15 14:11:24 INFO client.RMProxy: Connecting to ResourceManager at /
>> 0.0.0.0:8032
>> 13/11/15 14:11:24 INFO yarn.GiraphYarnClient: Got node report from ASM
>> for, nodeId=mbp-rvesse.home:49981, nodeAddressmbp-rvesse.home:8042,
>> nodeRackName/default-rack, nodeNumContainers0
>> 13/11/15 14:11:24 INFO yarn.GiraphYarnClient: Defaulting per-task heap
>> size to 1024MB.
>> 13/11/15 14:11:24 INFO yarn.GiraphYarnClient: Obtained new Application
>> ID: application_1384523910015_0004
>> 13/11/15 14:11:24 INFO Configuration.deprecation: mapred.job.id is
>> deprecated. Instead, use mapreduce.job.id
>> 13/11/15 14:11:24 INFO yarn.GiraphYarnClient: Set the environment for the
>> application master
>> 13/11/15 14:11:24 INFO yarn.GiraphYarnClient: Environment for AM
>> :{CLASSPATH=${CLASSPATH}:./*:$HADOOP_CONF_DIR:$HADOOP_COMMON_HOME/share/hadoop/common/*:$HADOOP_COMMON_HOME/share/hadoop/common/lib/*:$HADOOP_HDFS_HOME/share/hadoop/hdfs/*:$HADOOP_HDFS_HOME/share/hadoop/hdfs/lib/*:$HADOOP_YARN_HOME/share/hadoop/yarn/*:$HADOOP_YARN_HOME/share/hadoop/yarn/lib/*:$HADOOP_MAPRED_HOME/share/hadoop/mapreduce/*,$HADOOP_MAPRED_HOME/share/hadoop/mapreduce/lib/*}
>> 13/11/15 14:11:24 INFO yarn.GiraphYarnClient: buildLocalResourceMap ....
>> 13/11/15 14:11:24 INFO Configuration.deprecation: mapred.output.dir is
>> deprecated. Instead, use mapreduce.output.fileoutputformat.outputdir
>> 13/11/15 14:11:25 INFO yarn.YarnUtils: Registered file in LocalResources
>> ::
>> hdfs://localhost/user/rvesse/giraph_yarn_jar_cache/application_1384523910015_0004/giraph-conf.xml
>> 13/11/15 14:11:25 INFO yarn.GiraphYarnClient: LIB JARS
>> :/Users/rvesse/Documents/Work/Code/giraph/giraph-examples/target/giraph-examples-1.1.0-SNAPSHOT-for-hadoop-2.2.0-jar-with-dependencies.jar
>> 13/11/15 14:11:25 INFO yarn.YarnUtils: Class path name .
>> 13/11/15 14:11:25 INFO yarn.YarnUtils: base path checking .
>> 13/11/15 14:11:25 INFO yarn.YarnUtils: Class path name null
>> 13/11/15 14:11:25 INFO yarn.YarnUtils: base path checking null
>> 13/11/15 14:11:25 INFO yarn.YarnUtils: Class path name
>> /Users/rvesse/Documents/Apps/hadoop-2.2.0/etc/hadoop
>> 13/11/15 14:11:25 INFO yarn.YarnUtils: base path checking
>> /Users/rvesse/Documents/Apps/hadoop-2.2.0/etc/hadoop
>> 13/11/15 14:11:25 INFO yarn.YarnUtils: Class path name
>> /Users/rvesse/Documents/Apps/hadoop-2.2.0/share/hadoop/common/lib/*
>> 13/11/15 14:11:25 INFO yarn.YarnUtils: base path checking
>> /Users/rvesse/Documents/Apps/hadoop-2.2.0/share/hadoop/common/lib
>> 13/11/15 14:11:25 INFO yarn.YarnUtils: Class path name
>> /Users/rvesse/Documents/Apps/hadoop-2.2.0/share/hadoop/common/*
>> 13/11/15 14:11:25 INFO yarn.YarnUtils: base path checking
>> /Users/rvesse/Documents/Apps/hadoop-2.2.0/share/hadoop/common
>> 13/11/15 14:11:25 INFO yarn.YarnUtils: Class path name
>> /Users/rvesse/Documents/Apps/hadoop-2.2.0/share/hadoop/hdfs
>> 13/11/15 14:11:25 INFO yarn.YarnUtils: base path checking
>> /Users/rvesse/Documents/Apps/hadoop-2.2.0/share/hadoop/hdfs
>> 13/11/15 14:11:25 INFO yarn.YarnUtils: Class path name
>> /Users/rvesse/Documents/Apps/hadoop-2.2.0/share/hadoop/hdfs/lib/*
>> 13/11/15 14:11:25 INFO yarn.YarnUtils: base path checking
>> /Users/rvesse/Documents/Apps/hadoop-2.2.0/share/hadoop/hdfs/lib
>> 13/11/15 14:11:25 INFO yarn.YarnUtils: Class path name
>> /Users/rvesse/Documents/Apps/hadoop-2.2.0/share/hadoop/hdfs/*
>> 13/11/15 14:11:25 INFO yarn.YarnUtils: base path checking
>> /Users/rvesse/Documents/Apps/hadoop-2.2.0/share/hadoop/hdfs
>> 13/11/15 14:11:25 INFO yarn.YarnUtils: Class path name
>> /Users/rvesse/Documents/Apps/hadoop-2.2.0/share/hadoop/yarn/lib/*
>> 13/11/15 14:11:25 INFO yarn.YarnUtils: base path checking
>> /Users/rvesse/Documents/Apps/hadoop-2.2.0/share/hadoop/yarn/lib
>> 13/11/15 14:11:25 INFO yarn.YarnUtils: Class path name
>> /Users/rvesse/Documents/Apps/hadoop-2.2.0/share/hadoop/yarn/*
>> 13/11/15 14:11:25 INFO yarn.YarnUtils: base path checking
>> /Users/rvesse/Documents/Apps/hadoop-2.2.0/share/hadoop/yarn
>> 13/11/15 14:11:25 INFO yarn.YarnUtils: Class path name
>> /Users/rvesse/Documents/Apps/hadoop-2.2.0/share/hadoop/mapreduce/lib/*
>> 13/11/15 14:11:25 INFO yarn.YarnUtils: base path checking
>> /Users/rvesse/Documents/Apps/hadoop-2.2.0/share/hadoop/mapreduce/lib
>> 13/11/15 14:11:25 INFO yarn.YarnUtils: Class path name
>> /Users/rvesse/Documents/Apps/hadoop-2.2.0/share/hadoop/mapreduce/*
>> 13/11/15 14:11:25 INFO yarn.YarnUtils: base path checking
>> /Users/rvesse/Documents/Apps/hadoop-2.2.0/share/hadoop/mapreduce
>> 13/11/15 14:11:25 INFO yarn.YarnUtils: Class path name
>> /contrib/capacity-scheduler/*.jar
>> 13/11/15 14:11:25 INFO yarn.YarnUtils: base path checking
>> /contrib/capacity-scheduler
>> 13/11/15 14:11:25 INFO yarn.GiraphYarnClient:
>> ApplicationSumbissionContext for GiraphApplicationMaster launch container
>> is populated.
>> 13/11/15 14:11:25 INFO yarn.GiraphYarnClient: Submitting application to
>> ASM
>> 13/11/15 14:11:25 INFO impl.YarnClientImpl: Submitted application
>> application_1384523910015_0004 to ResourceManager at /0.0.0.0:8032
>> 13/11/15 14:11:25 INFO yarn.GiraphYarnClient: Got new appId after
>> submission :application_1384523910015_0004
>> 13/11/15 14:11:25 INFO yarn.GiraphYarnClient: GiraphApplicationMaster
>> container request was submitted to ResourceManager for job: Giraph:
>> org.apache.giraph.examples.SimpleShortestPathsComputation
>> 13/11/15 14:11:26 INFO yarn.GiraphYarnClient: Giraph:
>> org.apache.giraph.examples.SimpleShortestPathsComputation, Elapsed: 0.82
>> secs
>> 13/11/15 14:11:26 INFO yarn.GiraphYarnClient:
>> appattempt_1384523910015_0004_000001, State: ACCEPTED, Containers used: 1
>> 13/11/15 14:11:29 ERROR yarn.GiraphYarnClient: Giraph:
>> org.apache.giraph.examples.SimpleShortestPathsComputation reports FAILED
>> state, diagnostics show: Application application_1384523910015_0004 failed
>> 2 times due to AM Container for appattempt_1384523910015_0004_000002 exited
>> with  exitCode: 1 due to: Exception from container-launch:
>> org.apache.hadoop.util.Shell$ExitCodeException:
>> at org.apache.hadoop.util.Shell.runCommand(Shell.java:464)
>>  at org.apache.hadoop.util.Shell.run(Shell.java:379)
>> at
>> org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:589)
>>  at
>> org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:195)
>> at
>> org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:283)
>> at
>> org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:79)
>>  at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
>> at java.util.concurrent.FutureTask.run(FutureTask.java:138)
>>  at
>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918)
>>  at java.lang.Thread.run(Thread.java:695)
>>
>>
>> .Failing this attempt.. Failing the application.
>> 13/11/15 14:11:29 INFO yarn.GiraphYarnClient: Cleaning up HDFS
>> distributed cache directory for Giraph job.
>> 13/11/15 14:11:29 INFO yarn.GiraphYarnClient: Completed Giraph:
>> org.apache.giraph.examples.SimpleShortestPathsComputation: FAILED, total
>> running time: 0 minutes, 3 seconds.
>>
>> And I see the same error in the logs:
>>
>> Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/giraph/yarn/GiraphApplicationMaster
>> Caused by: java.lang.ClassNotFoundException: org.apache.giraph.yarn.GiraphApplicationMaster
>> 	at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
>> 	at java.security.AccessController.doPrivileged(Native Method)
>> 	at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
>> 	at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
>> 	at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
>> 	at java.lang.ClassLoader.loadClass(ClassLoader.java:247)
>>
>>
>> Any more ideas?
>>
>>
>> Rob
>>
>>
>> From: Mohammad Islam <mislam77@yahoo.com>
>> Reply-To: <user@giraph.apache.org>
>> Date: Thursday, 14 November 2013 20:36
>> To: "user@giraph.apache.org" <user@giraph.apache.org>
>> Subject: Re: Giraph and Hadoop 2.2.0
>>
>> I tried the following command few months back. I found you are not
>> passing some params.
>>
>> bin/hadoop --config etc/hadoop jar
>> /export/home/clei/giraph/giraph-1.0.0/giraph-examples/target/giraph-examples-1.?.0-for-hadoop-???-jar-with-dependencies.jar
>> org.apache.giraph.GiraphRunner
>> org.apache.giraph.examples.SimplePageRankVertexOpt -c
>> org.apache.giraph.combiner.DoubleSumCombiner -w 35 -yh 20480 -yj
>> giraph-examples-1.?.?-for-hadoop-2.??-jar-with-dependencies.jar -vif
>> org.apache.giraph.io.formats.JsonLongDoubleFloatDoubleVertexInputFormat -of
>> org.apache.giraph.io.formats.IdWithValueTextOutputFormat -vip
>> /user/me/input -op /user/me/output_35w_20G_80m_1
>>
>>
>>   On Thursday, November 14, 2013 3:22 AM, Rob Vesse <rvesse@dotnetrdf.org>
>> wrote:
>>  It's the same command line I showed earlier, just trying to run the
>> example application:
>>
>> mbp-rvesse:hadoop-2.2.0 rvesse$ bin/hadoop jar
>> /Users/rvesse/Documents/Work/Code/giraph/giraph-examples/target/giraph-examples-1.1.0-SNAPSHOT-for-hadoop-2.2.0-jar-with-dependencies.jar
>> org.apache.giraph.GiraphRunner
>> org.apache.giraph.examples.SimpleShortestPathsComputation -vif
>> org.apache.giraph.io.formats.JsonLongDoubleFloatDoubleVertexInputFormat
>> -vip /user/rvesse/tiny_graph.txt -vof
>> org.apache.giraph.io.formats.IdWithValueTextOutputFormat -op
>> /user/rvesse/shortestpaths/2 -w 1
>>
>> Rob
>>
>> From: Mohammad Islam <mislam77@yahoo.com>
>> Reply-To: <user@giraph.apache.org>
>> Date: Thursday, 14 November 2013 05:57
>> To: "user@giraph.apache.org" <user@giraph.apache.org>
>> Subject: Re: Giraph and Hadoop 2.2.0
>>
>> What are you running? PR?
>> What is the command line?
>>
>>
>>
>>   On Wednesday, November 13, 2013 8:30 AM, Rob Vesse <
>> rvesse@dotnetrdf.org> wrote:
>>  So I fixed the previous issue I was having which I reported and
>> submitted a patch as GIRAPH-795
>>
>> I then went ahead with installing and configuring my external Zookeeper
>> appropriately and tried running again but now the YARN application fails
>> and I find the following error in the logs:
>>
>> Exception in thread "main" java.lang.NoClassDefFoundError:
>>  org/apache/giraph/yarn/GiraphApplicationMaster
>> Caused by: java.lang.ClassNotFoundException: org.apache.giraph.yarn.GiraphApplicationMaster
>> 	at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
>> 	at java.security.AccessController.doPrivileged(Native Method)
>> 	at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
>> 	at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
>> 	at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
>> 	at java.lang.ClassLoader.loadClass(ClassLoader.java:247)
>>
>>
>> Anybody have idea what the cause of this one might be or how to fix it?
>>
>>
>> Cheers,
>>
>>
>> Rob
>>
>>
>> From: Rob Vesse <rvesse@dotnetrdf.org>
>> Reply-To: <user@giraph.apache.org>
>> Date: Wednesday, 13 November 2013 15:35
>> To: <user@giraph.apache.org>
>> Subject: Re: Giraph and Hadoop 2.2.0
>>
>> Mohammad
>>
>> This got me a build that at least attempts to start to run but not I see
>> the following:
>>
>> mbp-rvesse:hadoop-2.2.0 rvesse$ bin/hadoop jar
>> /Users/rvesse/Documents/Work/Code/giraph/giraph-examples/target/giraph-examples-1.1.0-SNAPSHOT-for-hadoop-2.2.0-jar-with-dependencies.jar
>> org.apache.giraph.GiraphRunner
>> org.apache.giraph.examples.SimpleShortestPathsComputation -vif
>> org.apache.giraph.io.formats.JsonLongDoubleFloatDoubleVertexInputFormat
>> -vip /user/rvesse/tiny_graph.txt -vof
>> org.apache.giraph.io.formats.IdWithValueTextOutputFormat -op
>> /user/rvesse/shortestpaths/2 -w 1
>> 2013-11-13 15:13:29.908 java[17402:1203] Unable to load realm info from
>> SCDynamicStore
>> 13/11/13 15:13:29 WARN util.NativeCodeLoader: Unable to load
>> native-hadoop library for your platform... using builtin-java classes where
>> applicable
>> 13/11/13 15:13:31 INFO utils.ConfigurationUtils: No edge input format
>> specified. Ensure your InputFormat does not require one.
>> 13/11/13 15:13:31 INFO utils.ConfigurationUtils: No edge output format
>> specified. Ensure your OutputFormat does not require one.
>> 13/11/13 15:13:31 INFO yarn.GiraphYarnClient: Final output path is:
>> hdfs://localhost/user/rvesse/shortestpaths/2
>> Exception in thread "main" java.lang.NullPointerException
>>  at
>> org.apache.giraph.yarn.GiraphYarnClient.checkJobLocalZooKeeperSupported(GiraphYarnClient.java:460)
>> at org.apache.giraph.yarn.GiraphYarnClient.run(GiraphYarnClient.java:116)
>>  at org.apache.giraph.GiraphRunner.run(GiraphRunner.java:96)
>> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
>>  at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:84)
>> at org.apache.giraph.GiraphRunner.main(GiraphRunner.java:126)
>>  at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>> at
>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>>  at
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>> at java.lang.reflect.Method.invoke(Method.java:597)
>>  at org.apache.hadoop.util.RunJar.main(RunJar.java:212)
>>
>> Which indirectly implies that I have some missing configuration to run
>> Giraph on YARN.  I don't think that should be a NPE and it looks to be a
>> trivial fix so I'll try and fix and send a pull request for that.
>>
>> The error message I should receive if the NPE wasn't happening is the
>> following:
>>
>> Giraph on YARN does not currentlysupport Giraph-managed ZK instances: use
>> a standalone ZooKeeper.
>>
>> What configuration options do I need to have set to point Giraph at an
>> external Zookeeper?
>>
>> Cheers,
>>
>> Rob
>>
>> From: Mohammad Islam <mislam77@yahoo.com>
>> Reply-To: <user@giraph.apache.org>
>> Date: Wednesday, 13 November 2013 11:15
>> To: "user@giraph.apache.org" <user@giraph.apache.org>
>> Subject: Re: Giraph and Hadoop 2.2.0
>>
>> Hi Rob,
>> Did you try with this command "mvn -Phadoop_yarn -Dhadoop.version=2.2.0
>> clean package -DskipTests"?
>> Regards,
>> Mohammad
>>
>>
>>   On Wednesday, November 13, 2013 2:55 AM, Rob Vesse <
>> rvesse@dotnetrdf.org> wrote:
>>  Hi all
>>
>> I'm having trouble getting Giraph to build and run against a Hadoop 2.2.0
>> installation.  I tried the hadoop_2.0.3 profile with no luck because I get
>> class cast errors when trying to run the examples.
>>
>> Is there a specific profile I should be building to support Hadoop 2.2.0
>> or is this not supported yet?
>>
>> Thanks,
>>
>> Rob
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>

Mime
View raw message