giraph-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Jeyendran Balakrishnan" ...@personaltube.com>
Subject Can Giraph handle graphs with very large number of edges per vertex?
Date Tue, 11 Sep 2012 06:23:36 GMT
I am trying to understand what kind of data Giraph holds in memory per
worker.
My questions in descending order of importance:
1. Does Giraph hold in memory exactly one vertex of data at a time, or does
it need to hold all the vertexes assigned to that worker?
2. Can Giraph handle vertexes with, a million edges per vertex?
   If not, at what order of magnitude does it break down? - 1000 edges, 10K
edges, 100K edges?... 
  (Of course, I understand that this depends upon the -Xmx value, so let's
say we fix a value of -Xmx8g).
3. Are there any limitations on the kind of objects that can be used as
vertices?
   Specifically, does Giraph assume that vertices are lightweight (eg,
integer vertex ID + simple Java primitive vertex values + collection of
out-edges),
   or can Giraph support heavyweight vertices (hold complex nested Java
objects in a vertex)?
4. More generally, what data is stored in memory, and what, if any, is
offloaded/spilled to disk?

Would appreciate any light the experts can throw on this.

On this note, I would like to mention that the presentations posted on the
Wiki explain what Giraph can do, and how to use it from  a coding
perspective, but there are no explanations of the design approach used, the
rationale behind the choices, and the software architecture. I feel that new
users can really benefit from a design  and architecture document, along the
lines of Hadoop and  Lucene. For folks who are considering whether or not to
use Giraph, this can be a big help. The only alternative today is to read
the source code, the burden of which might in itself be reason for folks not
to consider using Giraph. 
My 2c  :-)

Thanks a lot,
Jeyendran



Mime
View raw message