Mailing-List: contact dev-help@hbase.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@hbase.apache.org
Received-SPF: unknown (nike.apache.org: error in processing during lookup of
 mingma@ebay.com)
DomainKey-Signature: s=corp; d=ebay.com; c=nofws; q=dns;
  h=X-EBay-Corp:X-IronPort-AV:Received:Received:From:To:Date:
   Subject:Thread-Topic:Thread-Index:Message-ID:
   Accept-Language:Content-Language:X-MS-Has-Attach:
   X-MS-TNEF-Correlator:acceptlanguage:x-ems-proccessed:
   x-ems-stamp:Content-Type:MIME-Version:X-CFilter;
  b=jtKFYaWv8XwLQpMFPpgzZjMNU8HQGZXsJhjARWSRTduozz0z93zgeflZ
   iIWMs/4nxmKZYgt0uzROkPdsjAEa2PVcgmm9T95d7YkkX5HFCuCNJV89h
   PNRT9uiHxuJbReP;
From: "Ma, Ming" <mingma@ebay.com>
To: "dev@hbase.apache.org" <dev@hbase.apache.org>
Date: Thu, 23 Jun 2011 16:05:50 -0600
Subject: region assignment and HFile HDFS block locality
Thread-Topic: region assignment and HFile HDFS block locality
Thread-Index: Acwx8baVQFpCRigBQ9m4EfVnPR+d0g==
Message-ID: 
 <D10C58C9BA49DC40AA68056049009BF3642D7CB806@DEN-MEXMS-001.corp.ebay.com>
Accept-Language: en-US
Content-Language: en-US
acceptlanguage: en-US
Content-Type: multipart/alternative;
	boundary="_000_D10C58C9BA49DC40AA68056049009BF3642D7CB806DENMEXMS001co_"
MIME-Version: 1.0

--_000_D10C58C9BA49DC40AA68056049009BF3642D7CB806DENMEXMS001co_
Content-Type: text/plain; charset="us-ascii"
Content-Transfer-Encoding: quoted-printable

Normally, when we put hbase and HDFS in the same cluster ( e.g., region ser=
ver runs on the datenode ), we have a reasonably good data locality, as exp=
lained<http://www.larsgeorge.com/2010/05/hbase-file-locality-in-hdfs.html> =
by Lars. Also Work<https://issues.apache.org/jira/browse/HBASE-2896> has be=
en done by Jonathan to address the startup situation.

There are scenarios where regions can be on a different machine from the ma=
chines that hold the underlying HFile blocks, at least for some period of t=
ime. This will have performance impact on whole table scan operation and ma=
p reduce job during that time.


1.       After load balancer moves the region and before compaction (thus g=
enerate HFile on the new region server ) on that region, HDFS block can be =
remote.

2.       When a new machine is added, or removed, Hbase's region assignment=
 policy is different from HDFS's block reassignment policy.

3.       Even if there is no much hbase activity, HDFS can load balance HFi=
le blocks as other non-hbase applications push other data to HDFS.

Lots has been or will be done in load balancer, as summarized<http://zhihon=
gyu.blogspot.com/2011/04/load-balancer-in-hbase-090.html> by Ted. I am curi=
ous if HFile HDFS block locality should be used as another factor here.

Thanks.

Ming

--_000_D10C58C9BA49DC40AA68056049009BF3642D7CB806DENMEXMS001co_--