giraph-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Alessandro Presta" <alessan...@fb.com>
Subject Review Request: Move part of the graph out-of-core when memory is low
Date Mon, 16 Jul 2012 14:18:10 GMT

-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/5987/
-----------------------------------------------------------

Review request for giraph and Avery Ching.


Description
-------

I replaced the HashMap that stores partitions in a worker with a WorkerPartitionMap.
A WorkerPartitionMap has a normal in-memory map, and the ability to store entire partitions
to the local FS when memory is low.

In order to provide the normal views of the contents of a map, we operate lazily by loading
the out-of-core partitions as we iterate.
We always add the requested partition to the in-memory map (moving another one to disk to
make room) in order to allow modification.

The option "giraph.outOfCoreGraph" controls whether we use WorkerPartitionMap or a normal
HashMap as before.
"giraph.minFreeMemoryRatio" controls how much free memory we want to preserve out of the maximum
available memory for the program.
If out-of-core is enabled and the memory limit is exceeded, we start spilling partitions to
disk.

My main concern now is with input splitting: we may have to reorganize that phase so that
messages are not all accumulated in temporary in-memory storage before they are transferred
to the (disk-backed) WorkerPartitionMap.
See http://bit.ly/NwwFGI and subsequent comments for more context.


This addresses bug GIRAPH-249.
    https://issues.apache.org/jira/browse/GIRAPH-249


Diffs
-----

  http://svn.apache.org/repos/asf/giraph/trunk/src/main/java/org/apache/giraph/graph/BspService.java
1360711 
  http://svn.apache.org/repos/asf/giraph/trunk/src/main/java/org/apache/giraph/graph/BspServiceWorker.java
1360711 
  http://svn.apache.org/repos/asf/giraph/trunk/src/main/java/org/apache/giraph/graph/GiraphJob.java
1360711 
  http://svn.apache.org/repos/asf/giraph/trunk/src/main/java/org/apache/giraph/graph/WorkerPartitionMap.java
PRE-CREATION 
  http://svn.apache.org/repos/asf/giraph/trunk/src/test/java/org/apache/giraph/graph/TestWorkerPartitionMap.java
PRE-CREATION 

Diff: https://reviews.apache.org/r/5987/diff/


Testing
-------

- Unit test.
- "mvn verify" with different settings in order to trigger the out-of-core mechanism.
- Plan on doing some benchmarking.


Thanks,

Alessandro Presta


Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message