giraph-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From José Luis Larroque <larroques...@gmail.com>
Subject Re: RELATION BETWEEN THE NUMBER OF GIRAPH WORKERS AND THE PROBLEM SIZE
Date Wed, 22 Feb 2017 17:30:46 GMT
I remember that a good practice is using 1 worker per node, there are
several emails recommending this. It's the best way to use the maximum RAM
available in the cluster i believe.

Bye



-- 
*José Luis Larroque*
Analista Programador Universitario - Facultad de Informática - UNLP
Desarrollador Java y .NET  en LIFIA

2017-02-22 7:34 GMT-03:00 Ramesh Krishnan <ramesh.154089@gmail.com>:

> HI Ganesh,
>
> Recommendation is to increase the number of nodes with lesser ram size.
> Your number of executors depend on the CPU core hence, i would recommend
> using 60 GB RAM cpu's with 2 executors each for your use case.
>
> Thanks
> Ramesh
>
> On Wed, Feb 22, 2017 at 3:27 PM, Sai Ganesh Muthuraman <
> saiganeshpsn@gmail.com> wrote:
>
>> Hi,
>>
>> I am running a giraph application in the XSEDE comet cluster for graphs
>> of different sizes.
>> For a graph with 10,000 edges, I used about 8 workers on 2 nodes, each
>> node having 128GB RAM. My input file itself is just about 200KB.
>> But when I tried to increase the number of workers to 20 or more and the
>> number of nodes, the application takes infinite time and does not finish at
>> all.
>>
>> I have another graph data of size 50MB or so that has millions of edges.
>> If the number of workers is 2 or 3, I get this error
>> * java.lang.OutOfMemoryError: Java heap space*
>> If the number of workers is more, then the application doesn't end at all.
>> What is the best way to arrive at the number of workers and the number of
>> nodes, given the problem size? Is trial and error the only way?
>>
>>
>> Sai Ganesh
>>
>>
>>
>

Mime
View raw message