hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Raghu Angadi <rang...@yahoo-inc.com>
Subject Re: NN memory consumption on 0.20/0.21 with compressed pointers/
Date Thu, 20 Aug 2009 17:51:57 GMT

Suresh had made an spreadsheet for memory consumption.. will check.

A large portion of NN memory is taken by references. I would expect 
memory savings to be very substantial (same as going from 64bit to 
32bit), could be on the order of 40%.

The last I heard from Sun was that compressed pointers will be in very 
near future JVM (certainly JDK 1.6_x). It can use compressed pointers 
upto 32GB of heap.

I would expect runtime over head on NN would be minimal in practice.


Steve Loughran wrote:
> does anyone have any up to date data on the memory consumption per 
> block/file on the NN on a 64-bit JVM with compressed pointers?
> The best documentation on consumption is 
> http://issues.apache.org/jira/browse/HADOOP-1687 -I'm just wondering if 
> anyone has looked at the memory footprint on the latest Hadoop releases, 
> on those latest JVMs? -and which JVM the numbers from HADOOP-1687 came 
> from?
> Those compressed pointers (which BEA JRockit had for a while) save RAM 
> when the pointer references are within a couple of GB of the other refs, 
> and which are discussed in some papers
> http://rappist.elis.ugent.be/~leeckhou/papers/cgo06.pdf
> http://www.elis.ugent.be/~kvenster/papers/VenstermansKris_ORA.pdf
> sun's commentary is up here
> http://wikis.sun.com/display/HotSpotInternals/CompressedOops
> I'm just not sure what it means for the NameNode, and as there is no 
> sizeof() operator in Java, something that will take a bit of effort to 
> work out. From what I read of the Sun wiki, when you go compressed, 
> while your heap is <3-4GB, there is no decompress operation; once you go 
> above that there is a shift and an add, which is probably faster than 
> fetching another 32 bits from $L2 or main RAM. The result could be 
> -could be- that your NN takes up much less space on 64 bit JVMs than it 
> did before, but is no slower.
> Has anyone worked out the numbers yet?
> -steve

View raw message