giraph-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Lukas Nalezenec <>
Subject Re: Custom assignment of partitions to workers
Date Wed, 25 Mar 2015 19:54:09 GMT

There are two interfaces:
WorkerGraphPartitioner - Maps vertexes to partitions
MasterGraphPartitioner - Maps partitions to workers.
So you need custom MasterGraphPartitioner.
You dont need any external preprocessing step.


On 25.3.2015 19:51, Arjun Sharma wrote:
> Hi,
> I understand we can override the GraphPartitionerFactory class in 
> order to achieve custom partitioning of vertices over partitions. Is 
> there a way to do the same to enable assigning partitions to workers 
> in a custom way (e.g., partition n should be assigned to worker m)? 
> The reason is that it is more beneficial to have partitions that are 
> close to each other in terms of the graph structure placed close to 
> each other on the same worker to minimize network traffic. Otherwise, 
> benefits of overriding  GraphPartitionerFactory may be lost. Let us 
> assume that there is an external preprocessing step that outputs the 
> desired assignment, so can we materialize that assignment over Giraph 
> workers?
> Thanks!

View raw message