giraph-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Pascal Jäger <pas...@pascaljaeger.de>
Subject Using out of core messages
Date Thu, 24 Apr 2014 14:21:37 GMT
Hi all,

I am struggling with the settings to use out of core messages.
I have 3 nodes with 16 GB RAM each ( one master, two workers).

I ran into a java heap space OOM Error.

First question is: Where do I set the mapred.child.java-opts Options?
Do I need to add them via the "-ca  mapred.child….“ option or by using „-Dmapred.child…..“
I tried both, but nothing seems to work out.
I run it on a cloudera cluster, and when looking in the web frontend I see, that it only uses
3 GB of my 16 GB RAM.
Are those even the right options ?

giraph.maxMessagesInMemory - is it per worker? Or what exactly is counted here? and how does
it correlate to giraph.messagesBufferSize?

I am really lost right now. My graph has currently only 8000 nodes and 70000 edges.
During one step I need to send more than 15 000 000 messages and this is when I get the OOM
error.

I turned on the out of core messages feature without changing the above mentioned options
and my computation really slowed down.
I guess because it was writing 14 000 000 messages to disk

Hope you can help me.

Regards Pascal



Mime
View raw message