giraph-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Sai Ganesh Muthuraman" <saiganesh...@gmail.com>
Subject RELATION BETWEEN THE NUMBER OF GIRAPH WORKERS AND THE PROBLEM SIZE
Date Wed, 22 Feb 2017 09:57:10 GMT
Hi, I am running a giraph application in the XSEDE comet cluster for graphs of different sizes. 
For a graph with 10,000 edges, I used about 8 workers on 2 nodes, each node having 128GB RAM.
My input file itself is just about 200KB. 

But when I tried to increase the number of workers to 20 or more and the number of nodes,
the application takes infinite time and does not finish at all. 

I have another graph data of size 50MB or so that has millions of edges. If the number of
workers is 2 or 3, I get this error

 java.lang.OutOfMemoryError: Java heap space

If the number of workers is more, then the application doesn't end at all.

What is the best way to arrive at the number of workers and the number of nodes, given the
problem size? Is trial and error the only way? Sai Ganesh
Mime
View raw message