hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Nguyen Kien Trung <trung....@gmail.com>
Subject Re: How much RAMs needed...
Date Mon, 16 Jul 2007 02:07:49 GMT
Hi Peter,

I appreciate for the info. I'm afraid I'm not getting what you mean.
The issue I've encountered is i'm not able to start up the namenode due 
to out of memory error. Given that there are huge number of tiny files 
in datanodes.

Cheers,

Trung

Peter W. wrote:
> Trung,
>
> Using one machine (with 2GB RAM) and 300 input files
> I was able to successfully run:
>
> INFO mapred.JobClient:
>
> Map input records=10785802
> Map output records=10785802
> Map input bytes=1302175673
> Map output bytes=1101864522
> Reduce input groups=1244034
> Reduce input records=10785802
> Reduce output records=1050704
>
> Consolidating the files in your input
> directories might help.
>
> Peter W.
>
>
> On Jul 15, 2007, at 5:40 PM, Ted Dunning wrote:
>
>>
>> Are these really tiny files, or are you really storing 2M x 100MB = 
>> 200TB of
>> data? Or do you have more like 2M x 10KB = 20GB of data?
>>
>> Map-reduce and HDFS will generally work much better if you can 
>> arrange to
>> have relatively larger files.
>>
>>
>> On 7/15/07 8:04 AM, "erolagnab" <trung.n.k@gmail.com> wrote:
>>
>>>
>>> I have a HDFS with 2 datanodes and 1 namenode in 3 different 
>>> machines, 2G ram
>>> each.
>>> Datanode A contains around 700,000 blocks and Datanode B contains 
>>> 1,200,000+
>>> blocks, the namenode fails to start due to out of memory when trying 
>>> to add
>>> Datanode B into its rack. I have adjusted the java heap memory to 
>>> 1600MB
>>> which is the maxinum. But it still runs out of memory.
>>>
>>> AFAIK, namenode loads all blocks information into the memory. If so, 
>>> then is
>>> there anyway to estimate how much ram needed for a HDFS with given 
>>> number of
>>> blocks in each datanode?
>>
>
>


Mime
View raw message