hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Konstantin Shvachko <...@yahoo-inc.com>
Subject Re: Namenode BlocksMap on Disk
Date Wed, 26 Nov 2008 23:20:56 GMT
 > We can also try to mount the particular dir on ramfs

This could be an interesting project to replace the whole name-node
that is its FSDirectory part by ramfs.
If there are people interested in that kind of experiments we can discuss it.

 >>> 2) Besides possible slight degradation in performance, is there a
 >>> reason why the BlocksMap shouldn't or couldn't be stored on disk?

The degradation is expected to be substantial. But another reason is that it
will introduce a whole new layer in the name-node implementation responsible
of keeping the memory image in sync with the disk block map with caching etc.

As Raghu mentioned there was an idea of separating blockMap into a standalone
server, but currently there is no ongoing progress in this direction.

--Konstantin

Sagar Naik wrote:
> We can also try to mount the particular dir on ramfs and reduce the 
> performance degradation
> 
> -Sagar
> Billy Pearson wrote:
>> I would like to see something like this also I run 32bit servers so I 
>> am limited on how much memory I can use for heap. Besides just storing 
>> to disk I would like to see some sort of cache like a block cache that 
>> will cache parts the BlocksMap this would help reduce the hits to disk 
>> for lookups and still give us the ability to lower the memory 
>> requirement for the namenode.
>>
>> Billy
>>
>>
>> "Dennis Kubes" <kubes@apache.org> wrote in message 
>> news:492D3D15.7070401@apache.org...
>>> From time to time a message pops up on the mailing list about OOM 
>>> errors for the namenode because of too many files.  Most recently 
>>> there was a 1.7 million file installation that was failing.  I know 
>>> the simple solution to this is to have a larger java heap for the 
>>> namenode.  But the non-simple way would be to convert the BlocksMap 
>>> for the NameNode to be stored on disk and then queried and updated 
>>> for operations.  This would eliminate memory problems for large file 
>>> installations but also might degrade performance slightly.  Questions:
>>>
>>> 1) Is there any current work to allow the namenode to store on disk 
>>> versus is memory?  This could be a configurable option.
>>>
>>> 2) Besides possible slight degradation in performance, is there a 
>>> reason why the BlocksMap shouldn't or couldn't be stored on disk?
>>>
>>> I am willing to put forth the work to make this happen.  Just want to 
>>> make sure I am not going down the wrong path to begin with.
>>>
>>> Dennis
>>>
>>
>>
> 
> 

Mime
View raw message