hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From prasenjit mukherjee <prasen....@gmail.com>
Subject Re: NameNode per-block memory usage?
Date Wed, 18 Jan 2012 12:30:52 GMT
Does it mean that on an average 1 file has only 2 blocks ( with
replication=1 ) ?


On 1/18/12, M. C. Srivas <mcsrivas@gmail.com> wrote:
> Konstantin's paper
> http://www.usenix.org/publications/login/2010-04/openpdfs/shvachko.pdf
>
> mentions that on average a file consumes about 600 bytes of memory in the
> name-node (1 file object + 2 block objects).
>
> To quote from his paper (see page 9)
>
> ".. in order to store 100 million files (referencing 200 million blocks) a
> name-node should have at least 60GB of RAM. This matches observations on
> deployed clusters".
>
>
>
> On Tue, Jan 17, 2012 at 7:08 AM, Otis Gospodnetic <
> otis_gospodnetic@yahoo.com> wrote:
>
>> Hello,
>>
>> How much memory/JVM heap does NameNode use for each block?
>>
>> I've tried locating this in the FAQ and on search-hadoop.com, but
>> couldn't find a ton of concrete numbers, just these two:
>>
>> http://search-hadoop.com/m/RmxWMVyVvK1 - 150 bytes/block?
>> http://search-hadoop.com/m/O886P1VyVvK1 - 1 GB heap for every object?
>>
>> Thanks,
>> Otis
>>
>

-- 
Sent from my mobile device

Mime
View raw message