Mailing-List: contact issues-help@hbase.apache.org; run by ezmlm
Precedence: bulk
Date: Tue, 11 Oct 2016 20:41:20 +0000 (UTC)
From: "David Pope (JIRA)" <jira@apache.org>
To: issues@hbase.apache.org
Message-ID: <JIRA.13011462.1476211862000.797167.1476218480571@Atlassian.JIRA>
In-Reply-To: <JIRA.13011462.1476211862000@Atlassian.JIRA>
References: <JIRA.13011462.1476211862000@Atlassian.JIRA> <JIRA.13011462.1476211862607@arcas>
Subject: [jira] [Updated] (HBASE-16810) HBase Balancer throws
 ArrayIndexOutOfBoundsException when regionservers in /hbase/draining znode
 and unloaded
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
archived-at: Tue, 11 Oct 2016 20:41:22 -0000


     [ https://issues.apache.org/jira/browse/HBASE-16810?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

David Pope updated HBASE-16810:
-------------------------------
    Attachment: master.patch

The issue is that the wrong value from cluster.regionLocations is sent as the server index to cluster.getLocalityOfRegion(int region, int server) in LocalityCostFunction.cost().  Specifically, the index of the region's RegionLocation is being passed instead of its value, where the value is the server index.

As a result, the calculation is based on a "random" server rather than the server hosting the region.  So, this calculation is calculating the wrong value even when it doesn't throw an exception.

In the case when region servers are in draining, the draining servers are removed from the servers array and a "-1" is stored where it existed in cluster.RegionLocations.  If the server that is hosting the region is stored at an index of cluster.RegionLocations greater than the size of the servers array, cluster.getLocalityOfRegion(int region, int server) will throw an ArrayIndexOutOfBoundsException.

E.g., 

servers:
  [0] = "server0"
  [1] = "server1"
  [2] = "server2"

regions:
  [0] = "region0"
  [1] = "region1"
  [2] = "region2"

regionIndexToServerIndex:
  [0] = 2  // region0 is hosted on server2
  [1] = 0
  [3] = 1

RegionLocations:
  [0][0] = 1  // region0 has blocks on server1
  [0][1] = 0  
  [0][2] = -1  // region0 has blocks on a server in draining
  [0][3] = 2   // this is the matching entry, but 3 is used as the server index instead of 2 when calling getLocalityOfRegion(int region, int server) 

> HBase Balancer throws ArrayIndexOutOfBoundsException when regionservers in /hbase/draining znode and unloaded
> -------------------------------------------------------------------------------------------------------------
>
>                 Key: HBASE-16810
>                 URL: https://issues.apache.org/jira/browse/HBASE-16810
>             Project: HBase
>          Issue Type: Bug
>          Components: Balancer
>    Affects Versions: 2.0.0, 1.3.0
>            Reporter: Ashu Pachauri
>            Assignee: David Pope
>         Attachments: master.patch
>
>
> 1. Add a regionserver znode under /hbase/draining znode.
> 2. Use RegionMover to unload all regions from the regionserver.
> 3. Run balancer.
> {code}
> 16/09/21 14:17:33 ERROR ipc.RpcServer: Unexpected throwable object
> java.lang.ArrayIndexOutOfBoundsException: 75
>       at org.apache.hadoop.hbase.master.balancer.BaseLoadBalancer$Cluster.getLocalityOfRegion(BaseLoadBalancer.java:867)
>       at org.apache.hadoop.hbase.master.balancer.StochasticLoadBalancer$LocalityCostFunction.cost(StochasticLoadBalancer.java:1186)
>       at org.apache.hadoop.hbase.master.balancer.StochasticLoadBalancer.computeCost(StochasticLoadBalancer.java:521)
>       at org.apache.hadoop.hbase.master.balancer.StochasticLoadBalancer.balanceCluster(StochasticLoadBalancer.java:309)
>       at org.apache.hadoop.hbase.master.balancer.StochasticLoadBalancer.balanceCluster(StochasticLoadBalancer.java:264)
>       at org.apache.hadoop.hbase.master.HMaster.balance(HMaster.java:1339)
>       at org.apache.hadoop.hbase.master.MasterRpcServices.balance(MasterRpcServices.java:442)
>       at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:58555)
>       at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2268)
>       at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123)
>       at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188)
>       at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168)
> {code}


--
This message was sent by Atlassian JIRA
(v6.3.4#6332)