giraph-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jyotirmoy Sundi <>
Subject memory usage in giraph
Date Wed, 16 Oct 2013 21:22:35 GMT
    I am having data of around 4B Vertices and 13548192791 edges, say with
avg outgoing edge of 4, so total 4B*40 bytes= 160G , When I load this into
memory across 165 mappers each with 6G mem, all memory get occupied, I am
confused here as we have around 1TB space in memory. What is its thats
taking ~800G , is it the mapping information stored ? is there anyway of
doing profiling over it ?

Best Regards,
Jyotirmoy Sundi
Data Engineer,

San Francisco, CA 94158

View raw message