giraph-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From David Garcia <>
Subject Re: question about vertex instantiation location. . .
Date Fri, 10 Feb 2012 21:52:26 GMT
Ah, so, I think I would like to balance by vertices.  My main question is
that my graph starts with two vertices. . .I would like to specify more
than two mappers.  My job will end up creating around 100,000 vertices.  I
would like to make sure that these extra vertices will be evenly
distributed across all mappers (including the ones that don't have the
initial two vertices).  Does this make sense?  Does Giraph support this
out of the box, or do I need to add something?  Thx.


On 2/10/12 3:41 PM, "Avery Ching" <> wrote:

>By default, you are using the HashPartitionerFactory.  This will create
>the partitions ahead of time and balance them equally by count to the
>workers.  Therefore, assuming you have a uniform distribution across the
>VertexId space, the graph should be balanced across the workers evenly
>according the number of vertices.  If you look at PartitionBalancer, you
>can try to rebalance the graph if you like as it is running.  This is a
>bit experimental, but should work.  The choices for balancing are (no
>balancing, balance by edges or balance by vertices).
>Hope that helps,
>On 2/10/12 1:25 PM, David Garcia wrote:
>> Hey guys. . .I have a questions about "dynamic" vertex instantiation vis
>> the sendMsg(. . .) method.  I have a job that starts processing on a
>> sequenceFile with only two vertices in it.  Each vertex has information
>> it's value that tells it what vertices are adjacent to it.  The primary
>> reason I'm doing this is to avoid loading the entire graph into the job.
>> There are many vertices that won't do any processing (no need to load
>> them).  I would like to take my two vertices and "dynamically" build the
>> graph by sending messages.  So far, my experimentation shows that this
>> promising. . .but I have a question WRT load balancing for new vertex
>> instantiation.  When I call sendMsg(newVertexID), where will the vertex
>> instantiated?  If I specify 20 mappers (but with only two vertices in my
>> sequence file), obviously there is going to be at least one mapper
>> a vertex.  Is it possible that sendMsg(newVertexID) will be instantiated
>> on an empty mapper?  I would like this. . .for load balancing purposes.
>> -david

View raw message