hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From James Seigel <ja...@tynt.com>
Subject Re: NameNode heapsize
Date Fri, 10 Jun 2011 12:16:49 GMT
Are you using compressed pointers?

Sent from my mobile. Please excuse the typos.

On 2011-06-10, at 5:33 AM, "siuon@ugcv.com" <siuon@ugcv.com> wrote:

> Dear all,
>
> I'm looking for ways to improve the namenode heap size usage of a 800-node 10PB testing
Hadoop cluster that stores
> around 30 million files.
>
> Here's some info:
>
> 1 x namenode:     32GB RAM, 24GB heap size
> 800 x datanode:   8GB RAM, 13TB hdd
>
> *33050825 files and directories, 47708724 blocks = 80759549 total. Heap Size is 22.93
GB / 22.93 GB (100%) *
>
> From the cluster summary report, it seems the heap size usage is always full but couldn't
drop, do you guys know of any ways
> to reduce it ? So far I don't see any namenode OOM errors so it looks memory assigned
for the namenode process is (just)
> enough. But i'm curious which factors would account for the full use of heap size ?
>
> Regards,
> On

Mime
View raw message