hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Doug Cutting <cutt...@apache.org>
Subject Re: Namenode BlocksMap on Disk
Date Wed, 26 Nov 2008 18:08:37 GMT
Dennis Kubes wrote:
> 2) Besides possible slight degradation in performance, is there a reason 
> why the BlocksMap shouldn't or couldn't be stored on disk?

I think the assumption is that it would be considerably more than slight 
degradation.  I've seen the namenode benchmarked at over 50,000 opens 
per second.  If file data is on disk, and the namespace is considerably 
bigger than RAM, then a seek would be required per access.  At 
10MS/seek, that would give only 100 opens per second, or 500x slower. 
Flash storage today peaks at around 5k seeks/second.

For smaller clusters the namenode might not need to be able to perform 
50k opens/second, but for larger clusters we do not want the namenode to 
become a bottleneck.


View raw message