Mailing-List: contact hdfs-dev-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
Reply-To: hdfs-dev@hadoop.apache.org
Received-SPF: pass (athena.apache.org: domain of dhruba@gmail.com designates
 209.85.161.48 as permitted sender)
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:date:message-id:subject:from:to
         :cc:content-type;
        b=TCottwWrXBRf71mZ/owLSHt+PITX0fKj4LPBitsfZAqfeCta3Pubo1QE1HJUK05chj
         8Ofak9Em/18Eiz1ycfUkiXibvOOnmRBHsdSQt0nzuEem0DuNDUd1wtbnMZg5K9Y6qXt1
         jrGPwRBsdYc/EO7wETbGtNBa66qOtWziTTH3c=
MIME-Version: 1.0
In-Reply-To: <BANLkTi=2ss0xiwnSQ0HgkiXBwRkC=HUcOw@mail.gmail.com>
References: <C9D6FF6E.19BFC%sureshms@yahoo-inc.com>
	<4DB5E962.2000206@apache.org>
	<39D2BF2A-D3B9-4A21-AF6F-DCBCA04EFEFF@yahoo-inc.com>
	<BANLkTi=2ss0xiwnSQ0HgkiXBwRkC=HUcOw@mail.gmail.com>
Date: Tue, 26 Apr 2011 21:27:30 -0700
Message-ID: <BANLkTim0zu6X6gNT9XwpB1iYzcT_hGhfGw@mail.gmail.com>
Subject: Re: [Discuss] Merge federation branch HDFS-1052 into trunk
From: Dhruba Borthakur <dhruba@gmail.com>
To: hdfs-dev@hadoop.apache.org
Cc: sradia@yahoo-inc.com, Doug Cutting <cutting@apache.org>
Content-Type: multipart/alternative; boundary=00151747658cde8eb604a1dedb4c

--00151747658cde8eb604a1dedb4c
Content-Type: text/plain; charset=ISO-8859-1

I feel that making the datanode talk to multiple namenodes is very valuable,
especially when there is plenty of storage available on a single datanode
machine (think 24 TB to 36 TB) and a single namenode does not have enough
memory to hold all file metadata for such a large cluster in memory.

This is a feature that we are in dire need of, and could put it to good use
starting "yesterday"!

thanks,
dhruba

On Tue, Apr 26, 2011 at 5:59 PM, Konstantin Boudnik <cos@apache.org> wrote:

> Sanjay,
>
> I assume the outlined changes won't an earlier version of HDFS from
> upgrads to the federation version, right?
>
> Cos
>
> On Tue, Apr 26, 2011 at 17:26, Sanjay Radia <sradia@yahoo-inc.com> wrote:
> >
> > Changes to the code base
> >  - The fundamental code change is to extend the notion of block id to now
> > include a block pool id.
> > - The  NN had little change, the protocols did change to include the
> block
> > pool id.
> > - The DN code did change. Each data structure is now indexed by the block
> > pool id -- while this is a code change, it is architecturally very simple
> > and low risk.
> > - We also did a fair amount of cleanup of threads used to send block
> reports
> > - while it was not strictly necessary to do the cleanup we took the extra
> > effort to pay the technical debt. As Dhruba recently noted, adding
> support
> > to send block reports to primary and secondary NN for HA will be now much
> > easier to do.
>


-- 
Connect to me at http://www.facebook.com/dhruba

--00151747658cde8eb604a1dedb4c--