HI Ganesh,
Recommendation is to increase the number of nodes with lesser ram size.
Your number of executors depend on the CPU core hence, i would recommend
using 60 GB RAM cpu's with 2 executors each for your use case.
Thanks
Ramesh
On Wed, Feb 22, 2017 at 3:27 PM, Sai Ganesh Muthuraman <
saiganeshpsn@gmail.com> wrote:
> Hi,
>
> I am running a giraph application in the XSEDE comet cluster for graphs of
> different sizes.
> For a graph with 10,000 edges, I used about 8 workers on 2 nodes, each
> node having 128GB RAM. My input file itself is just about 200KB.
> But when I tried to increase the number of workers to 20 or more and the
> number of nodes, the application takes infinite time and does not finish at
> all.
>
> I have another graph data of size 50MB or so that has millions of edges.
> If the number of workers is 2 or 3, I get this error
> * java.lang.OutOfMemoryError: Java heap space*
> If the number of workers is more, then the application doesn't end at all.
> What is the best way to arrive at the number of workers and the number of
> nodes, given the problem size? Is trial and error the only way?
>
>
> Sai Ganesh
>
>
>
