giraph-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Matthew Laird <lai...@sfu.ca>
Subject Re: Yarn application failure
Date Tue, 15 Oct 2013 19:50:07 GMT
So I've made some progress on this issue, and I'm hoping someone like 
Eli might have some insight after watching his great presentation on 
giraph and yarn... I'm receiving:

Error: Could not find or load main class 
org.apache.giraph.yarn.GiraphApplicationMaster

I'm guessing this has something to do with the -yj option and ensuring 
the jar is distributed to all the workers, am I looking in the correct 
direction or does anyone have advice on solving this issue?

Thanks.

On 13-10-14 04:55 PM, Matthew Laird wrote:
> After much googling I finally pieced together how to run in pure yarn
> mode, first, for any future googlers here's the two command lines I used
> to build and run giraph head:
>
> mvn -Dhadoop.version=2.0.3-alpha -Phadoop_yarn -DskipTests clean install
>
> $HADOOP_HOME/bin/hadoop jar
> $GIRAPH_HOME/giraph-examples/target/giraph-examples-1.1.0-SNAPSHOT-for-hadoop-2.0.3-alpha-jar-with-dependencies.jar
> org.apache.giraph.GiraphRunner -Dgiraph.zkList=zookeeper1:7000
> org.apache.giraph.examples.SimpleShortestPathsComputation -vif
> org.apache.giraph.io.formats.JsonLongDoubleFloatDoubleVertexInputFormat
> -vip /in/tiny_graph.txt -vof
> org.apache.giraph.io.formats.IdWithValueTextOutputFormat -op
> /user/hduser/output/shortestpaths -w 4
>
> (where -Dgiraph.zkList=zookeeper1:7000 is a comma separated list of
> zookeeper quorum members)
>
> Now on to my query... I'm wondering about suggestions on where to look
> for debugging information, upon running I received:
>
> 13/10/14 16:49:24 WARN util.NativeCodeLoader: Unable to load
> native-hadoop library for your platform... using builtin-java classes
> where applicable
> 13/10/14 16:49:24 INFO utils.ConfigurationUtils: No edge input format
> specified. Ensure your InputFormat does not require one.
> 13/10/14 16:49:24 INFO utils.ConfigurationUtils: No edge output format
> specified. Ensure your OutputFormat does not require one.
> 13/10/14 16:49:24 INFO yarn.GiraphYarnClient: Final output path is:
> hdfs://x007:9000/user/hduser/output/shortestpaths
> 13/10/14 16:49:24 INFO service.AbstractService:
> Service:org.apache.hadoop.yarn.client.YarnClientImpl is inited.
> 13/10/14 16:49:24 INFO service.AbstractService:
> Service:org.apache.hadoop.yarn.client.YarnClientImpl is started.
> 13/10/14 16:49:24 INFO yarn.GiraphYarnClient: Defaulting per-task heap
> size to 1024MB.
> 13/10/14 16:49:24 INFO yarn.GiraphYarnClient: Obtained new Application
> ID: application_1381087048629_0004
> 13/10/14 16:49:24 WARN conf.Configuration: mapred.job.id is deprecated.
> Instead, use mapreduce.job.id
> 13/10/14 16:49:24 WARN conf.Configuration: mapred.output.dir is
> deprecated. Instead, use mapreduce.output.fileoutputformat.outputdir
> 13/10/14 16:49:25 INFO yarn.YarnUtils: Registered file in
> LocalResources: giraph-conf.xml
> 13/10/14 16:49:25 INFO yarn.GiraphYarnClient:
> ApplicationSumbissionContext for GiraphApplicationMaster launch
> container is populated.
> 13/10/14 16:49:25 INFO client.YarnClientImpl: Submitted application
> application_1381087048629_0004 to ResourceManager at
> x007/192.168.10.117:8040
> 13/10/14 16:49:25 INFO yarn.GiraphYarnClient: GiraphApplicationMaster
> container request was submitted to ResourceManager for job: Giraph:
> org.apache.giraph.examples.SimpleShortestPathsComputation
> 13/10/14 16:49:26 INFO yarn.GiraphYarnClient: Giraph:
> org.apache.giraph.examples.SimpleShortestPathsComputation, Elapsed: 0.84
> secs
> 13/10/14 16:49:26 INFO yarn.GiraphYarnClient:
> appattempt_1381087048629_0004_000001, State: ACCEPTED, Containers used: 1
> 13/10/14 16:49:27 ERROR yarn.GiraphYarnClient: Giraph:
> org.apache.giraph.examples.SimpleShortestPathsComputation reports FAILED
> state, diagnostics show: Application application_1381087048629_0004
> failed 1 times due to AM Container for
> appattempt_1381087048629_0004_000001 exited with exitCode: 1 due to:
> .Failing this attempt.. Failing the application.
> 13/10/14 16:49:27 INFO yarn.GiraphYarnClient: Cleaning up HDFS
> distributed cache directory for Giraph job.
> 13/10/14 16:49:27 INFO yarn.GiraphYarnClient: Completed Giraph:
> org.apache.giraph.examples.SimpleShortestPathsComputation: FAILED, total
> running time: 0 minutes, 1 seconds.
> lairdm@x007:/opt/giraph$ $HADOOP_HOME/bin/hadoop jar
> $GIRAPH_HOME/giraph-examples/target/giraph-examples-1.1.0-SNAPSHOT-for-hadoop-2.0.3-alpha-jar-with-dependencies.jar
> org.apache.giraph.GiraphRunner -Dgiraph.zkList=zookeeper1:7000
> org.apache.giraph.examples.SimpleShortestPathsComputation -vif
> org.apache.giraph.io.formats.JsonLongDoubleFloatDoubleVertexInputFormat
> -vip /in/tiny_graph.txt -vof
> org.apache.giraph.io.formats.IdWithValueTextOutputFormat -op
> /user/hduser/output/shortestpaths -w 4
> 13/10/14 16:52:15 WARN util.NativeCodeLoader: Unable to load
> native-hadoop library for your platform... using builtin-java classes
> where applicable
> 13/10/14 16:52:16 INFO utils.ConfigurationUtils: No edge input format
> specified. Ensure your InputFormat does not require one.
> 13/10/14 16:52:16 INFO utils.ConfigurationUtils: No edge output format
> specified. Ensure your OutputFormat does not require one.
> 13/10/14 16:52:16 INFO yarn.GiraphYarnClient: Final output path is:
> hdfs://x007:9000/user/hduser/output/shortestpaths
> 13/10/14 16:52:16 INFO service.AbstractService:
> Service:org.apache.hadoop.yarn.client.YarnClientImpl is inited.
> 13/10/14 16:52:16 INFO service.AbstractService:
> Service:org.apache.hadoop.yarn.client.YarnClientImpl is started.
> 13/10/14 16:52:16 INFO yarn.GiraphYarnClient: Defaulting per-task heap
> size to 1024MB.
> 13/10/14 16:52:16 INFO yarn.GiraphYarnClient: Obtained new Application
> ID: application_1381087048629_0005
> 13/10/14 16:52:16 WARN conf.Configuration: mapred.job.id is deprecated.
> Instead, use mapreduce.job.id
> 13/10/14 16:52:16 WARN conf.Configuration: mapred.output.dir is
> deprecated. Instead, use mapreduce.output.fileoutputformat.outputdir
> 13/10/14 16:52:16 INFO yarn.YarnUtils: Registered file in
> LocalResources: giraph-conf.xml
> 13/10/14 16:52:17 INFO yarn.GiraphYarnClient:
> ApplicationSumbissionContext for GiraphApplicationMaster launch
> container is populated.
> 13/10/14 16:52:17 INFO client.YarnClientImpl: Submitted application
> application_1381087048629_0005 to ResourceManager at
> x007/192.168.10.117:8040
> 13/10/14 16:52:17 INFO yarn.GiraphYarnClient: GiraphApplicationMaster
> container request was submitted to ResourceManager for job: Giraph:
> org.apache.giraph.examples.SimpleShortestPathsComputation
> 13/10/14 16:52:18 INFO yarn.GiraphYarnClient: Giraph:
> org.apache.giraph.examples.SimpleShortestPathsComputation, Elapsed: 0.84
> secs
> 13/10/14 16:52:18 INFO yarn.GiraphYarnClient:
> appattempt_1381087048629_0005_000001, State: ACCEPTED, Containers used: 1
> 13/10/14 16:52:19 ERROR yarn.GiraphYarnClient: Giraph:
> org.apache.giraph.examples.SimpleShortestPathsComputation reports FAILED
> state, diagnostics show: Application application_1381087048629_0005
> failed 1 times due to AM Container for
> appattempt_1381087048629_0005_000001 exited with exitCode: 1 due to:
> .Failing this attempt.. Failing the application.
> 13/10/14 16:52:19 INFO yarn.GiraphYarnClient: Cleaning up HDFS
> distributed cache directory for Giraph job.
> 13/10/14 16:52:19 INFO yarn.GiraphYarnClient: Completed Giraph:
> org.apache.giraph.examples.SimpleShortestPathsComputation: FAILED, total
> running time: 0 minutes, 1 seconds.
>
> Is there a log somewhere of why it failed or a way to get more detailed
> output from the AM? Getting closer, I'm doing little jumps for joy.
>
> Thanks, and great work getting this package put together in the first
> place!
>

-- 
Matthew Laird
Lead Software Developer, Bioinformatics
Brinkman Laboratory
Simon Fraser University, Burnaby, BC, Canada

Mime
View raw message