hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Brian Bockelman <bbock...@cse.unl.edu>
Subject Re: Datanode high memory usage
Date Tue, 01 Sep 2009 11:59:38 GMT
Hey Mafish,

If you are getting 1-2m blocks on a single datanode, you'll have many  
other problems - especially with regards to periodic block reports.

With regards to memory, have you tried the compressed pointers JDK  
option (we saw great benefits on the NN)?  Java is incredibly hard to  
get a straight answer from with regards to memory.  You need to  
perform a GC first manually - the actual usage is the amount it  
reports used post-GC.  You can get these details by using JMX.

Brian

On Sep 1, 2009, at 4:08 AM, Mafish Liu wrote:

> Both NameNode and DataNode will be affected by number of files  
> greatly.
> In my test, almost 60% memory are used in datanodes while storing 1m
> files, and the value reach 80% with 2m files.
> My test best is with 5 nodes, 1 namenode and 4 datanodes. All nodes
> have 2GB memory and replication is 3.
>
> 2009/9/1 Stas Oskin <stas.oskin@gmail.com>:
>> Hi.
>>
>> 2009/9/1 Mafish Liu <mafish@gmail.com>
>>
>>> Did you have many small files in your system?
>>>
>>>
>> Yes, quite plenty.
>>
>> But this should influence the Namenode, and not the Datanode,  
>> correct?
>>
>> Regards.
>>
>
>
>
> -- 
> Mafish@gmail.com


Mime
View raw message