giraph-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Alessandro Presta" <>
Subject Review Request: Move part of the graph out-of-core when memory is low
Date Mon, 16 Jul 2012 14:18:10 GMT

This is an automatically generated e-mail. To reply, visit:

Review request for giraph and Avery Ching.


I replaced the HashMap that stores partitions in a worker with a WorkerPartitionMap.
A WorkerPartitionMap has a normal in-memory map, and the ability to store entire partitions
to the local FS when memory is low.

In order to provide the normal views of the contents of a map, we operate lazily by loading
the out-of-core partitions as we iterate.
We always add the requested partition to the in-memory map (moving another one to disk to
make room) in order to allow modification.

The option "giraph.outOfCoreGraph" controls whether we use WorkerPartitionMap or a normal
HashMap as before.
"giraph.minFreeMemoryRatio" controls how much free memory we want to preserve out of the maximum
available memory for the program.
If out-of-core is enabled and the memory limit is exceeded, we start spilling partitions to

My main concern now is with input splitting: we may have to reorganize that phase so that
messages are not all accumulated in temporary in-memory storage before they are transferred
to the (disk-backed) WorkerPartitionMap.
See and subsequent comments for more context.

This addresses bug GIRAPH-249.




- Unit test.
- "mvn verify" with different settings in order to trigger the out-of-core mechanism.
- Plan on doing some benchmarking.


Alessandro Presta

  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message