hadoop-hdfs-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Brian Bockelman <bbock...@cse.unl.edu>
Subject Re: Merging Namenode Federation feature (HDFS-1052) to trunk
Date Mon, 21 Mar 2011 23:25:07 GMT

On Mar 21, 2011, at 6:08 PM, Sanjay Radia wrote:

> On Mar 14, 2011, at 10:57 AM, Sanjay Radia wrote:
>> On Mar 12, 2011, at 8:43 AM, Allen Wittenauer wrote:
>>> 	To me, this series of changes looks like it is going to make
>>> running a grid much much harder for very little benefit.  In
>>> particular, I don't see the difference between running multiple NN/
>>> DN combinations verses running federation, especially with client
>>> side mount tables in play.
>> Main difference between independent HDFS clusters and HDFS federation
>> is that in federation one can shares the storage of the DNs and the DNs.
>> There is a very detailed document that describes this on the Jira.
>> If you are running a single NN and you don't need the scaling then
>> running and managing hadoop is for all practical purposes unchanged.
>> sanjay
> Allen, not sure if I explained the difference above.
> Base on the discussion we had at the Hug, I want to clarify a few things
> In federation the NNs and the DNs are part of  a cluster. It is not as if a data node
is willing to store blocks for any NN anywhere in the data center.
> We still expect a data center to have multiple hadoop clusters each with a set of data
nodes and each cluster with 1 or more NNs.
> A DN stores block for only ONE cluster.

A few questions:
- Do we have a clear definition for a cluster?
- With the above definition, is it an error if not all DNs belong to the same set of NNs?
- With the working definition of a cluster, what namespace guarantees are given to clients?

The reason I ask is not because I oppose the idea of federations, but rather am curious of
about the terminology and how it's 'advertised' to the user.  I rather like the design; it
has similar ideas to a NSF project I've seen (http://www.reddnet.org/).

> You had asked about how one debugs a corrupt file or corrupt block.
> In the old world a file's inode contains the block ids of its blocks. There is also a
mapping from block id to block location (ie which DN).
> In the federated hdfs, each block is identified by a longer block id, called the extended
block id= blockPool Id + block id.
> A block pool is owned by only ONE NN.
> Hence if you are trying to locate a block then you map the extended block id to the block
location (ie DN) - this is the same as before, except that the identifier
> of the block is merely longer.
> If you are trying to debug from the point of view of the DN:
> In federated HDFS, the blocks stored in the DN are segregated in directories by the blockPool
> The block pool id can be mapped to a NN since each Block pool has only  ONE owner.
> Hence to map from a block to a particular NN is easy - the first part of the Block's
longer identifier  will tell you which NN owns that block.

This sounds good.


View raw message