giraph-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From André Kelpe <efeshundert...@googlemail.com>
Subject Re: Saving checkpoints?
Date Sun, 12 Aug 2012 10:57:31 GMT
Hi Vishal,

you can control the checkpoint frequency with the setting
"giraph.checkpointFrequency" in your JobConfiguration. The default is
set to 0 right now, meaning no checkpoints are made. You should def.
check out the GiraphJob [0] code, where all these tuning knobs are
documented.

--André

[0] https://github.com/apache/giraph/blob/trunk/src/main/java/org/apache/giraph/graph/GiraphJob.java#L308

2012/8/11 Vishal Patel <write2vishal@gmail.com>:
> Hi,
>
> How do I specify the interval for saving checkpoints? When working with
> Amazon's Elastic Mapreduce on a large number of workers (> 80 workers, 40 x
> m1.xlarge machines), sometimes there is RPC communication errors and
> Zookeeper waits on that worker for a while before timing out and killing the
> job all together.
>
> As my graph and number of workers is becoming larger I would like to learn
> how to save it since that extra cost might be well worth it-- say every 50
> supersteps. Here is the command I use currently, how should I modify it.
>
> hadoop jar giraph-0.2-SNAPSHOT-jar-with-dependencies.jar
> org.apache.giraph.GiraphRunner
> org.apache.giraph.examples.ConnectedComponentsVertex \
> --inputFormat org.apache.giraph.examples.IntIntNullIntTextInputFormat \
> --inputPath giraph_in/adj_list.txt \
> --outputFormat
> org.apache.giraph.examples.VertexWithComponentTextOutputFormat \
> --outputPath giraph_out
> --combiner org.apache.giraph.examples.MinimumIntCombiner
> --workers 95
>
> Also, how do I restart from a specific checkpoint. The help for the
> GiraphRunner class did not have instructions on this.
>
> Thank you!
>
> Vishal
>
>

Mime
View raw message