hbase-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "David Pope (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (HBASE-16810) HBase Balancer throws ArrayIndexOutOfBoundsException when regionservers in /hbase/draining znode and unloaded
Date Tue, 11 Oct 2016 20:41:20 GMT

     [ https://issues.apache.org/jira/browse/HBASE-16810?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

David Pope updated HBASE-16810:
-------------------------------
    Attachment: master.patch

The issue is that the wrong value from cluster.regionLocations is sent as the server index
to cluster.getLocalityOfRegion(int region, int server) in LocalityCostFunction.cost().  Specifically,
the index of the region's RegionLocation is being passed instead of its value, where the value
is the server index.

As a result, the calculation is based on a "random" server rather than the server hosting
the region.  So, this calculation is calculating the wrong value even when it doesn't throw
an exception.

In the case when region servers are in draining, the draining servers are removed from the
servers array and a "-1" is stored where it existed in cluster.RegionLocations.  If the server
that is hosting the region is stored at an index of cluster.RegionLocations greater than the
size of the servers array, cluster.getLocalityOfRegion(int region, int server) will throw
an ArrayIndexOutOfBoundsException.

E.g., 

servers:
  [0] = "server0"
  [1] = "server1"
  [2] = "server2"

regions:
  [0] = "region0"
  [1] = "region1"
  [2] = "region2"

regionIndexToServerIndex:
  [0] = 2  // region0 is hosted on server2
  [1] = 0
  [3] = 1

RegionLocations:
  [0][0] = 1  // region0 has blocks on server1
  [0][1] = 0  
  [0][2] = -1  // region0 has blocks on a server in draining
  [0][3] = 2   // this is the matching entry, but 3 is used as the server index instead of
2 when calling getLocalityOfRegion(int region, int server) 

> HBase Balancer throws ArrayIndexOutOfBoundsException when regionservers in /hbase/draining
znode and unloaded
> -------------------------------------------------------------------------------------------------------------
>
>                 Key: HBASE-16810
>                 URL: https://issues.apache.org/jira/browse/HBASE-16810
>             Project: HBase
>          Issue Type: Bug
>          Components: Balancer
>    Affects Versions: 2.0.0, 1.3.0
>            Reporter: Ashu Pachauri
>            Assignee: David Pope
>         Attachments: master.patch
>
>
> 1. Add a regionserver znode under /hbase/draining znode.
> 2. Use RegionMover to unload all regions from the regionserver.
> 3. Run balancer.
> {code}
> 16/09/21 14:17:33 ERROR ipc.RpcServer: Unexpected throwable object
> java.lang.ArrayIndexOutOfBoundsException: 75
>       at org.apache.hadoop.hbase.master.balancer.BaseLoadBalancer$Cluster.getLocalityOfRegion(BaseLoadBalancer.java:867)
>       at org.apache.hadoop.hbase.master.balancer.StochasticLoadBalancer$LocalityCostFunction.cost(StochasticLoadBalancer.java:1186)
>       at org.apache.hadoop.hbase.master.balancer.StochasticLoadBalancer.computeCost(StochasticLoadBalancer.java:521)
>       at org.apache.hadoop.hbase.master.balancer.StochasticLoadBalancer.balanceCluster(StochasticLoadBalancer.java:309)
>       at org.apache.hadoop.hbase.master.balancer.StochasticLoadBalancer.balanceCluster(StochasticLoadBalancer.java:264)
>       at org.apache.hadoop.hbase.master.HMaster.balance(HMaster.java:1339)
>       at org.apache.hadoop.hbase.master.MasterRpcServices.balance(MasterRpcServices.java:442)
>       at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:58555)
>       at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2268)
>       at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123)
>       at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188)
>       at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168)
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message