hadoop-hdfs-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Dhruba Borthakur <dhr...@gmail.com>
Subject Re: Regarding design of HDFS
Date Tue, 06 Sep 2011 04:52:53 GMT
My answers inline.

1. Why does namenode store the blockmap (block to datanode mapping) in the
main memory for all the files, even those that are not used?

The block to datanode mapping is needed for two reasons: when a client wants
to read a file, the namenode has to tell the client the locations of the
blocks that make up the file. Also, when a datanode dies, the namenode has
to quickly find the blocks that resided on that datanode so that it can
re-replicate those blocks.

2. Why cant namenode move out a part of the blockmap from main memory to a
secondary storage device, when free space in main memory becomes scarce (
due to large number of files) ?
3. Why cant the blockmap be constructed when a file is requested (by a
client) and then be cached for later accesses?

Both of the above can be done if needed. But when there is a better way to
scale, why do this? Please look at my comments below.

> The only way to avoid this is to make the data much more cacheable and to
> have a viable cache coherency strategy.  Cache coherency at the meta-data
> level is difficult.  Cache coherency at the block level is also difficult
> (but not as difficult) because many blocks get moved for balance purposes.
I would argue that a federation-model is much more scalable, elegant and
easier to maintain. It takes a very well-oiled building block like the
NameNode and allows you to use multitudes of them in a single  This is
already part of HDFS trunk code base.


View raw message