giraph-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Eli Reisman (JIRA)" <>
Subject [jira] [Commented] (GIRAPH-249) Move part of the graph out-of-core when memory is low
Date Sun, 15 Jul 2012 18:03:34 GMT


Eli Reisman commented on GIRAPH-249:

Yes, I had to drop GIRAPH-247 and start GIRAPH-256 because of some confusion I had over the
naming scheme (or lack thereof) in that same path from InputSplit to final loading of vertices
into the workerPartitionMap. From what I can tell, the vertices are stored in temp partitions
in BspServiceWorker during load from input splits, and are sent in bursts over netty to their
final homes (even when that is on the local worker, will file a separate jira/patch for this
monday I think) and this takes advantage of a single code path to get vertices to their new
owners whether it is INPUT_SUPERSTEP or a dynamic repartitioning during calculation super
steps. When the collections of vertices arrives at their new home they are again placed in
a Partition container (might file jira to eliminate all the middleman containers for memory
saving as well, since they are not sent on the wire in Partitions) and then combined into
existing partitions in workerPartitionMap using the correct partition ID from a list of ID's
that partition is the owner of according to info the master gives it at the beginning of each
super step.

As to outgoing partitions when they are first taken from InputSplit, they are stored in TEMPORARY
(this fooled me on GIRAPH-247 for a while) Parition objects in inputSplitCache or something
like that, and only sent out on the wire when they are a reasonable size to their real home.
this is what GIRAPH-256 improves, and  have had VERY good results testing it this weekend.
Anyway the inputSplitCache for that Partition object is cleared for refilling as the vertexReader
continues, and the vertices are sent over the wire as a Collection<BasicVertex> which
is also confusing since the request type is SendPartitionRequest or something like that.

So: no "real Partitions" are created or used for anything but temp  storage until vertices
arrive at the worker who will actually host them.

I can test your patch monday with metrics. Thing to remember though is this: it might be a
hard sell to introduce disk caching at all stages of the computation, because Giraph is based
on Pregel which is strictly in-memory processing. Once we leave behind the Pregel model entirely
we might be in danger of "feature creep" that could make it hard to sell new adopters on Giraph,
or even to explain to them how it works and why its a correct approach. I think being able
to claim we are a "BSP implementation following the Pregel model" will help adoption long
term just as Hadoop can claim to be a "MapReduce implementation with distributed filesystem
based on GFS" and everyone knew just what that was and was not.

Either way, great work, this is a lot to take on! Thanks again!

> Move part of the graph out-of-core when memory is low
> -----------------------------------------------------
>                 Key: GIRAPH-249
>                 URL:
>             Project: Giraph
>          Issue Type: Improvement
>            Reporter: Alessandro Presta
>            Assignee: Alessandro Presta
>         Attachments: GIRAPH-249.patch, GIRAPH-249.patch, GIRAPH-249.patch, GIRAPH-249.patch,
> There has been some talk about Giraph's scaling limitations due to keeping the whole
graph and messages in RAM.
> We need to investigate methods to fall back to disk when running out of memory, while
gracefully degrading performance.
> This issue is for graph storage. Messages should probably be a separate issue, although
the interplay between the two is crucial.
> We should also discuss what are our primary goals here: completing a job (albeit slowly)
instead of failing when the graph is too big, while still encouraging memory optimizations
and high-memory clusters; or restructuring Giraph to be as efficient as possible in disk mode,
making it almost a standard way of operating.

This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:!default.jspa
For more information on JIRA, see:


View raw message