giraph-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Eli Reisman <apache.mail...@gmail.com>
Subject Re: LocalJobRunner & examples
Date Sun, 10 Nov 2013 20:08:26 GMT
There's a patch that was never applied because the YARN profile has moved
to Hadoop 2.2.x GA compatibility, but it will help you run on
Hadoop-2.0.5-alpha if you want. Search it on the JIRA site, it creates
build profiles for 2.0.4 and 2.0.5 alpha releases.

I think when you set workers greater than 1, you are confusing the cluster
because it thinks you are in LocalRunnerMode. There are configurations you
must set in your *-site.xml files for your Hadoop install if you do not
wish to run in LocalMode. I'm note sure if Giraph at present understands
the difference between local mode with multiple-tasks or actual cluster
mode with many tasks.



On Mon, Oct 14, 2013 at 6:34 AM, Wei Zhang <weiz@us.ibm.com> wrote:

> Hi Matthew,
>
> It is my understanding that to run LocalJobRunner, you need to change your
> Hadoop configuration a bit.
>
> That is what I did:
> (1) in core-site.xml, comment out the fs.default.name property
> (2) in mapred-site.xml, comment out the mapred.job.tracker property
>
> Hope it helps.
>
> There are probably other smarter ways of doing this, and I would like to
> know as well.
>
> Wei
>
>
>
> [image: Inactive hide details for Matthew Laird ---10/13/2013 04:57:50
> PM---I've been trying to work through the Quick Start examples a]Matthew
> Laird ---10/13/2013 04:57:50 PM---I've been trying to work through the
> Quick Start examples and adapting  it to my local installation,
>
> From: Matthew Laird <lairdm@sfu.ca>
> To: user@giraph.apache.org,
> Date: 10/13/2013 04:57 PM
> Subject: LocalJobRunner & examples
> ------------------------------
>
>
>
> I've been trying to work through the Quick Start examples and adapting
> it to my local installation, I'm running Hadoop 2.0.5-alpha across 4
> machines.
>
> I've done the checkout from github
>
> sudo git clone https://github.com/apache/giraph.git
>
> Applied the patch to ZooKeeperManager
>
> http://www.mail-archive.com/user@giraph.apache.org/msg00945/check.diff
>
> Fixed the catching of the exception in
> giraph-core/src/main/java/org/apache/giraph/job/HaltApplicationUtils.java
>
> Then building with
>
> mvn clean install -Phadoop_2.0.3 -DskipTests
>
> Then when I go to run the examples, I set -w 4 because I am running a
> split master setup.  Unfortunately this leads to:
>
> lairdm@x007:/opt/giraph$ $HADOOP_HOME/bin/hadoop jar
> $GIRAPH_HOME/giraph-examples/target/giraph-examples-1.1.0-SNAPSHOT-for-hadoop-2.0.3-alpha-jar-with-dependencies.jar
>
> org.apache.giraph.GiraphRunner
> org.apache.giraph.examples.SimpleShortestPathsComputation -vif
> org.apache.giraph.io.formats.JsonLongDoubleFloatDoubleVertexInputFormat
> -vip /in/tiny_graph.txt -vof
> org.apache.giraph.io.formats.IdWithValueTextOutputFormat -op
> /user/hduser/output/shortestpaths -w 4
> 13/10/13 13:50:48 WARN util.NativeCodeLoader: Unable to load
> native-hadoop library for your platform... using builtin-java classes
> where applicable
> 13/10/13 13:50:48 INFO utils.ConfigurationUtils: No edge input format
> specified. Ensure your InputFormat does not require one.
> 13/10/13 13:50:48 INFO utils.ConfigurationUtils: No edge output format
> specified. Ensure your OutputFormat does not require one.
> 13/10/13 13:50:49 WARN conf.Configuration: mapreduce.job.counters.limit
> is deprecated. Instead, use mapreduce.job.counters.max
> 13/10/13 13:50:49 WARN conf.Configuration: mapred.job.map.memory.mb is
> deprecated. Instead, use mapreduce.map.memory.mb
> 13/10/13 13:50:49 WARN conf.Configuration: mapred.job.reduce.memory.mb
> is deprecated. Instead, use mapreduce.reduce.memory.mb
> 13/10/13 13:50:49 WARN conf.Configuration:
> mapred.map.tasks.speculative.execution is deprecated. Instead, use
> mapreduce.map.speculative
> 13/10/13 13:50:49 WARN conf.Configuration:
> mapreduce.user.classpath.first is deprecated. Instead, use
> mapreduce.job.user.classpath.first
> 13/10/13 13:50:49 WARN conf.Configuration: mapred.map.max.attempts is
> deprecated. Instead, use mapreduce.map.maxattempts
> 13/10/13 13:50:49 INFO job.GiraphJob: run: Since checkpointing is
> disabled (default), do not allow any task retries (setting
> mapred.map.max.attempts = 0, old value = 4)
> 13/10/13 13:50:49 WARN conf.Configuration: mapred.job.tracker is
> deprecated. Instead, use mapreduce.jobtracker.address
> Exception in thread "main" java.lang.IllegalArgumentException:
> checkLocalJobRunnerConfiguration: When using LocalJobRunner, must have
> only one worker since only 1 task at a time!
>         at
>
> org.apache.giraph.job.GiraphJob.checkLocalJobRunnerConfiguration(GiraphJob.java:159)
>         at org.apache.giraph.job.GiraphJob.run(GiraphJob.java:233)
>         at org.apache.giraph.GiraphRunner.run(GiraphRunner.java:94)
>         at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
>         at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:84)
>         at org.apache.giraph.GiraphRunner.main(GiraphRunner.java:124)
>         at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>         at
>
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
>         at
>
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>         at java.lang.reflect.Method.invoke(Method.java:601)
>         at org.apache.hadoop.util.RunJar.main(RunJar.java:212)
>
> I'm not sure what parameters I need to give giraph to signal not to use
> LocalJobRunner.  Does anyone have a suggestion?
>
> Thanks.
>
>
>

Mime
View raw message