Mailing-List: contact user-help@hbase.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@hbase.apache.org
Received-SPF: pass (nike.apache.org: domain of lars.george@gmail.com
 designates 209.85.214.41 as permitted sender)
Content-Type: text/plain; charset=iso-8859-1
Mime-Version: 1.0 (Apple Message framework v1278)
Subject: Re: Data locality in HBase
From: Lars George <lars.george@gmail.com>
In-Reply-To: 
 <CAMaDncLBrNgdQ8f7u3DoRAe=3PfSTsUgUvibJHsDvtc02Gfwew@mail.gmail.com>
Date: Fri, 15 Jun 2012 10:21:46 +0200
Content-Transfer-Encoding: quoted-printable
Message-Id: <C211D872-F4E1-44F2-B3FE-5CA72D673352@gmail.com>
References: 
 <CAMaDncLBrNgdQ8f7u3DoRAe=3PfSTsUgUvibJHsDvtc02Gfwew@mail.gmail.com>
To: user@hbase.apache.org

Hi Ben,

See inline...

On Jun 15, 2012, at 6:56 AM, Ben Kim wrote:

> Hi,
>=20
> I've been posting questions in the mailing-list quiet often lately, =
and
> here goes another one about data locality
> I read the excellent blog post about data locality that Lars George =
wrote
> at http://www.larsgeorge.com/2010/05/hbase-file-locality-in-hdfs.html
>=20
> I understand data locality in hbase as locating a region in a =
region-server
> where most of its data blocks reside.

The opposite is happening, i.e. the region server process triggers for =
all data it writes to be located on the same physical machine.=20

> So that way fast data access is guranteed when running a MR because =
each
> map/reduce task is run for each region in the tasktracker where the =
region
> co-locates.

Correct.

> But what if the data blocks of the region are evenly spread over =
multiple
> region-servers?

This will not happen, unless the original server fails. Then the region =
is moved to another that now needs to do a lot of remote reads over the =
network. This is way there is work being done to allow for custom =
placement policies in HDFS. That way you can store the entire region and =
all copies as complete units on three data nodes. In case of a failure =
you can then move the region to one of the two copies. This is not =
available yet though, but it is being worked on (so I heard).

> Does a MR task has to remotely access the data blocks from other
> regionservers?

For the above failure case, it would be the region server accessing the =
remote data, yes.

> How good is hbase locating datablocks where a region resides?

That is again the wrong way around. HBase has no clue as to where blocks =
reside, nor does it know that the file system in fact uses separate =
blocks. HBase stores files, HDFS does the block magic underneath the =
hood, and transparent to HBase.

> Also is it correct to say that if i set smaller data block size data
> locality gets worse, and if data block size gets bigger  data locality =
gets
> better.

This is not applicable here, I am assuming this stems from the above =
confusion about which system is handling the blocks, HBase or HDFS. See =
above.

HTH,
Lars

>=20
> Best regards,
> --=20
>=20
> *Benjamin Kim*
> *benkimkimben at gmail*