giraph-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Eli Reisman (JIRA)" <>
Subject [jira] [Commented] (GIRAPH-249) Move part of the graph out-of-core when memory is low
Date Thu, 12 Jul 2012 17:21:35 GMT


Eli Reisman commented on GIRAPH-249:

I've been looking at all this stuff too for my internship. For Jakob's fantastic metrics patch
GIRAPH-232, I have learned most of the memory danger is during INPUT_SUPERSTEP when InputSplits
become partitions and start to get moved around. Whether a worker is sending out partitions
as it builds them doesn't help us because that same worker is also receiving partitions of
its own from other workers that read a particular InputSplit! So this is the real memory danger
we've seen.

So...what about spilling to disk only when a worker is low on memory during INPUT_SUPERSTEP,
and at the end of it reloading them when the pressure of processing splits is done and the
pressure of mutating the graph and message passing begins. This is when the "partitions in
RAM" piece becomes such a win anyway. And best of all, no partitions are being mutated once
they are constructed and sent to their assigned worker during INPUT_SUPERSTEP, so any spilled
to disk will not need to return.

I have found using GIRAPH-247 to even out the lumpiness of large social graph data, the partition
sizes are already WAY more balanced than using the old vertex-centric way, so that also might
play well with this idea.

What do you think? If you try the metrics patch, you will see what I'm talking about with
the INPUT_SUPERSTEP, its dramatic with a large input data set.

> Move part of the graph out-of-core when memory is low
> -----------------------------------------------------
>                 Key: GIRAPH-249
>                 URL:
>             Project: Giraph
>          Issue Type: Improvement
>            Reporter: Alessandro Presta
>            Assignee: Alessandro Presta
>         Attachments: GIRAPH-249.patch
> There has been some talk about Giraph's scaling limitations due to keeping the whole
graph and messages in RAM.
> We need to investigate methods to fall back to disk when running out of memory, while
gracefully degrading performance.
> This issue is for graph storage. Messages should probably be a separate issue, although
the interplay between the two is crucial.
> We should also discuss what are our primary goals here: completing a job (albeit slowly)
instead of failing when the graph is too big, while still encouraging memory optimizations
and high-memory clusters; or restructuring Giraph to be as efficient as possible in disk mode,
making it almost a standard way of operating.

This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:!default.jspa
For more information on JIRA, see:


View raw message