giraph-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Vishal Patel <write2vis...@gmail.com>
Subject Saving checkpoints?
Date Fri, 10 Aug 2012 23:10:33 GMT
Hi,

How do I specify the interval for saving checkpoints? When working with
Amazon's Elastic Mapreduce on a large number of workers (> 80 workers, 40 x
m1.xlarge machines), sometimes there is RPC communication errors and
Zookeeper waits on that worker for a while before timing out and killing
the job all together.

As my graph and number of workers is becoming larger I would like to learn
how to save it since that extra cost might be well worth it-- say every 50
supersteps. Here is the command I use currently, how should I modify it.

hadoop jar giraph-0.2-SNAPSHOT-jar-with-dependencies.jar
org.apache.giraph.GiraphRunner
org.apache.giraph.examples.ConnectedComponentsVertex \
--inputFormat org.apache.giraph.examples.IntIntNullIntTextInputFormat \
--inputPath giraph_in/adj_list.txt \
--outputFormat
org.apache.giraph.examples.VertexWithComponentTextOutputFormat \
--outputPath giraph_out
--combiner org.apache.giraph.examples.MinimumIntCombiner
--workers 95

Also, how do I restart from a specific checkpoint. The help for the
GiraphRunner class did not have instructions on this.

Thank you!

Vishal

Mime
View raw message