giraph-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Sai Ganesh Muthuraman" <saiganesh...@gmail.com>
Subject Re: RELATION BETWEEN THE NUMBER OF GIRAPH WORKERS AND THE PROBLEM SIZE
Date Thu, 02 Mar 2017 10:01:09 GMT
Hi Jose,

I went through the container logs and found that the following error was happening
java.lang.OutOfMemoryError: Java heap space
This was probably causing the missing chosen workers error.

This happens only when the graph size exceeds more than 50k vertices and 100k edges.
I enabled out of core messaging, out of core computation and heap size is 8GB which is reasonable
given that I have 128 GB RAM in every node. 
I tried increasing the number of workers and the number of nodes to 6. Still the same result. 
This happens in superstep 2 itself.
Any suggestions?
Sai Ganesh


On Feb 27, 2017, at 10:27 PM, José Luis Larroque <user@giraph.apache.org> wrote:

> Could be a lot of different reasons. Memory problems, algorithm problems, etc.
> 
> I recommend you to focus in reach the logs instead of guessing why the worker's are dying.
Maybe you are looking in the wrong place, maybe you can access to them though web ui instead
of command line.
> 
> From terminal, doing yarn logs -applicationId "id" doing will b
Mime
View raw message