giraph-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Mirko Kämpf <mirko.kae...@cloudera.com>
Subject Re: Intermediate output
Date Sat, 18 Jan 2014 12:01:04 GMT
Hi Sebastian,

I would solve this within the worker, which takes the number of the
superstep to create
a snapshot in an individual subdirectory to store a snapshot of the graph
which consists of
all individual partitions. Finally you just merge all partition files to
one graph representation.
It is not the same thing like using an OutpuFormat, I think. The
subdirectorys should be created
at the beginn of the super step before the workers start processing the
data.
I did not implement this until now, but I work on the same or even a
comparable requirement.

Best wishes
Mirko

On Sat, Jan 18, 2014 at 10:02 AM, Sebastian Schelter <ssc@apache.org> wrote:

> Hi,
>
> Did we have a way to write out the state of the graph after each
> superstep? I have an algorithm that requires this and I don't want to
> buffer the intermediate results in memory until the algorithm finishes.
>
> --sebastian
>



--

Mime
View raw message