Mailing-List: contact user-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@hadoop.apache.org
Received-SPF: pass (nike.apache.org: domain of psybers@gmail.com designates
 209.85.223.176 as permitted sender)
MIME-Version: 1.0
Sender: psybers@gmail.com
In-Reply-To: 
 <CAPcDmSswK9vXHy-MakkMNAOGXKTwvi425bFk1F5bLLAL68ex8Q@mail.gmail.com>
References: 
 <CAMiz3FMm3MCcG1hnWegG0op56XQrdw2TJh9FTpUnS8Mf_+QAdg@mail.gmail.com>
	<CAPcDmSssVR919m4biB1FjTXvRmqeMgN3wZm4E5M26prXoWQ+=w@mail.gmail.com>
	<CAMiz3FNbR5szuQ0VkY5cOfntqihdyajg3noXkeAn5RRw8nC75w@mail.gmail.com>
	<CAPcDmSswK9vXHy-MakkMNAOGXKTwvi425bFk1F5bLLAL68ex8Q@mail.gmail.com>
Date: Wed, 29 Aug 2012 14:58:03 -0500
Message-ID: 
 <CAMiz3FPr8-0oCEiMZ2yRgYxu5EYp+i0iUz192DgQ6UOhDci=KA@mail.gmail.com>
Subject: Re: HBase and MapReduce data locality
From: Robert Dyer <rdyer@iastate.edu>
To: N Keywal <nkeywal@gmail.com>
Cc: user@hadoop.apache.org
Content-Type: text/plain; charset=ISO-8859-1

Ah thanks for that link.  I missed it while browsing the docs.  The
link from there to this blog post

  http://www.larsgeorge.com/2010/05/hbase-file-locality-in-hdfs.html

really answers my questions! :-)

On Wed, Aug 29, 2012 at 2:38 AM, N Keywal <nkeywal@gmail.com> wrote:
> Inline. Just a set of "you're right" :-).
> It's documented here:
> http://hbase.apache.org/book.html#regions.arch.locality
>
> On Wed, Aug 29, 2012 at 8:06 AM, Robert Dyer <rdyer@iastate.edu> wrote:
>>
>> Ok but does that imply that only 1 of your compute nodes is promised
>> to have all of the data for any given row?  The blocks will replicate,
>> but they don't necessarily all replicate to the same nodes right?
>
>
> Right.
>
>>
>> So if I have say 2 column families (cf1, cf2) and there is 2 physical
>> files on the HDFS for those (per region) then those files are created
>> on one datanode (dn1) which will have all blocks local to that node.
>
>
> Yes. Nit: datanodes don't "see" files, only blocks. But the logic remains
> the same.
>
>>
>> Once it replicates those blocks 2 more times by default, isn't it
>> possible the blocks for cf1 will go to dn2, dn3 while the blocks for
>> cf2 goes to dn4, dn5?
>
>
> Yes, it's possible (and even likely).