hadoop-hdfs-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Dhruba Borthakur <dhr...@gmail.com>
Subject Re: [Discuss] Merge federation branch HDFS-1052 into trunk
Date Wed, 27 Apr 2011 04:27:30 GMT
I feel that making the datanode talk to multiple namenodes is very valuable,
especially when there is plenty of storage available on a single datanode
machine (think 24 TB to 36 TB) and a single namenode does not have enough
memory to hold all file metadata for such a large cluster in memory.

This is a feature that we are in dire need of, and could put it to good use
starting "yesterday"!


On Tue, Apr 26, 2011 at 5:59 PM, Konstantin Boudnik <cos@apache.org> wrote:

> Sanjay,
> I assume the outlined changes won't an earlier version of HDFS from
> upgrads to the federation version, right?
> Cos
> On Tue, Apr 26, 2011 at 17:26, Sanjay Radia <sradia@yahoo-inc.com> wrote:
> >
> > Changes to the code base
> >  - The fundamental code change is to extend the notion of block id to now
> > include a block pool id.
> > - The  NN had little change, the protocols did change to include the
> block
> > pool id.
> > - The DN code did change. Each data structure is now indexed by the block
> > pool id -- while this is a code change, it is architecturally very simple
> > and low risk.
> > - We also did a fair amount of cleanup of threads used to send block
> reports
> > - while it was not strictly necessary to do the cleanup we took the extra
> > effort to pay the technical debt. As Dhruba recently noted, adding
> support
> > to send block reports to primary and secondary NN for HA will be now much
> > easier to do.

Connect to me at http://www.facebook.com/dhruba

  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message