Hi, I am running a giraph application in the XSEDE comet cluster for graphs of different sizes.
For a graph with 10,000 edges, I used about 8 workers on 2 nodes, each node having 128GB RAM.
My input file itself is just about 200KB.
But when I tried to increase the number of workers to 20 or more and the number of nodes,
the application takes infinite time and does not finish at all.
I have another graph data of size 50MB or so that has millions of edges. If the number of
workers is 2 or 3, I get this error
java.lang.OutOfMemoryError: Java heap space
If the number of workers is more, then the application doesn't end at all.
What is the best way to arrive at the number of workers and the number of nodes, given the
problem size? Is trial and error the only way? Sai Ganesh
