giraph-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Stefan Beskow <Stefan.Bes...@sas.com>
Subject Run SimpleShortestPathsVertex sample application using multiple workers
Date Mon, 24 Feb 2014 05:35:36 GMT
Hi.

I'm trying to run Giraph on Hadoop 2.0.0-cdh4.2.0 using a cluster with 60 nodes. When I run
the sample application org.apache.giraph.examples.SimpleShortestPathsVertex with just 1 worker
it works fine, but when I specify more than 1 worker it throws exception java.lang.IllegalArgumentException:
checkLocalJobRunnerConfiguration as shown below. Is there a way to pass a command line parameter
to Giraph so that it doesn't use the local job runner or do I need to update any of the Hadoop
configuration files for this to work?

Here is the command I use to run sample application with 2 workers:
hadoop jar giraph-examples-1.0.0-for-hadoop-2.0.0-cdh4.2.0-jar-with-dependencies.jar org.apache.giraph.GiraphRunner
-Dgiraph.zkList=rdcgrd001.unx.sas.com:2181 -libjars giraph-examples-1.0.0-for-hadoop-2.0.0-cdh4.2.0-jar-with-dependencies.jar
org.apache.giraph.examples.SimpleShortestPathsVertex -vif org.apache.giraph.io.formats.JsonLongDoubleFloatDoubleVertexInputFormat
-vip /user/stbesk/input/tiny_graph.txt -of org.apache.giraph.io.formats.IdWithValueTextOutputFormat
-op /user/stbesk/output/shortestpathsC2 -ca SimpleShortestPathsVertex.source=2 -w 2 -ca giraph.SplitMasterWorker=true

Here is the exception:
14/02/24 00:20:23 INFO utils.ConfigurationUtils: No edge input format specified. Ensure your
InputFormat does not require one.
14/02/24 00:20:23 INFO utils.ConfigurationUtils: Setting custom argument [SimpleShortestPathsVertex.source]
to [2] in GiraphConfiguration
14/02/24 00:20:23 INFO utils.ConfigurationUtils: Setting custom argument [giraph.SplitMasterWorker]
to [true] in GiraphConfiguration
14/02/24 00:20:23 WARN job.GiraphConfigurationValidator: Output format vertex index type is
not known
14/02/24 00:20:23 WARN job.GiraphConfigurationValidator: Output format vertex value type is
not known
14/02/24 00:20:23 WARN job.GiraphConfigurationValidator: Output format edge value type is
not known
14/02/24 00:20:23 INFO job.GiraphJob: run: Since checkpointing is disabled (default), do not
allow any task retries (setting mapred.map.max.attempts = 0, old value = 4)
Exception in thread "main" java.lang.IllegalArgumentException: checkLocalJobRunnerConfiguration:
When using LocalJobRunner, must have only one worker since only 1 task at a time!
        at org.apache.giraph.job.GiraphJob.checkLocalJobRunnerConfiguration(GiraphJob.java:151)
        at org.apache.giraph.job.GiraphJob.run(GiraphJob.java:225)
        at org.apache.giraph.GiraphRunner.run(GiraphRunner.java:94)
        at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
        at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:84)
        at org.apache.giraph.GiraphRunner.main(GiraphRunner.java:124)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
        at java.lang.reflect.Method.invoke(Method.java:597)
        at org.apache.hadoop.util.RunJar.main(RunJar.java:208)

Appreciate any help.

Thanks.
Stefan



Mime
View raw message