hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From lars hofhansl <la...@apache.org>
Subject Re: Determining regions with low HDFS locality index
Date Fri, 26 Dec 2014 20:43:13 GMT
There should be logic that attempts to restore the regions on the region servers that had them
last.
Note that the master can only assign regions to region server that have reported in. For that
reason the master waits a bit (4.5s by default) for region servers to report in after a master
start before it starts assigning regions.Maybe in your case that time is too short? You can
also configure the master to wait for a certain number of region server to report in.
If after you checked that it is still not working, could you file a jira outlining the details
and steps to reproduce?

In any case, if the master has to assign the regions to a subset of the region servers it
has no choice but to break locality. Then when the remaining region servers sign in in 0.94
there is no logic to maintain locality when the cluster is balanced. In 0.98 the stochastic
balancer uses locality as one of its parameters - although I have personally seen issues with
that that I still need to investigate.
-- Lars

      From: Rahul Ravindran <rahulrv@yahoo.com.INVALID>
 To: "user@hbase.apache.org" <user@hbase.apache.org> 
 Sent: Thursday, December 25, 2014 11:37 PM
 Subject: Determining regions with low HDFS locality index
   
Hi,   When an Hbase RS goes down(possibly because of hardware issues etc), the regions get
moved off that machine to other Region Servers. However, since the new region servers do not
have the backing HFiles, data locality for the newly transitioned regions is not great and
hence some of our jobs are a lot slower on these regions. Is there an API for me to determine
the regions within a RS which are responsible for low HDFS locality, for which I could trigger
a compaction to improve locality?
I took a look at HDFSBlocksDistribution from which I can determine the RS with low HDFS locality.
But, going from the RS level to the specific region which is responsible, seems harder. I
could try to look at the backing hfiles and determine locality using HDFS, but that seems
roundabout. Any suggestions?
I am running Hbase 0.94.15 with CDH 4.6
~Rahul. 

  
Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message