hbase-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Ted Yu (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HBASE-17565) StochasticLoadBalancer may incorrectly skip balancing due to skewed multiplier sum
Date Fri, 03 Feb 2017 17:09:51 GMT

    [ https://issues.apache.org/jira/browse/HBASE-17565?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15851743#comment-15851743
] 

Ted Yu commented on HBASE-17565:
--------------------------------

Here was excerpt from test output for TestStochasticLoadBalancer#testLargeCluster (with patch
v2):
{code}
2017-02-03 16:13:16,986 INFO  [main] balancer.StochasticLoadBalancer(362): start StochasticLoadBalancer.balancer,
initCost=107.0, functionCost=RegionCountSkewCostFunction : (500.0, 0.2); PrimaryRegionCountSkewCostFunction
: (500.0, 0.0); MoveCostFunction : (7.0, 0.0); LocalityCostFunction : (0.0, 0.0); TableSkewCostFunction
: (35.0, 0.2); RegionReplicaHostCostFunction : (100000.0, 0.0); RegionReplicaRackCostFunction
: (10000.0, 0.0); ReadRequestCostFunction : (5.0, 0.0); WriteRequestCostFunction : (5.0, 0.0);
MemstoreSizeCostFunction : (5.0, 0.0); StoreFileCostFunction : (5.0, 0.0);
2017-02-03 16:13:26,988 DEBUG [main] balancer.StochasticLoadBalancer(418): Finished computing
new load balance plan.  Computation took 10006ms to try 1000000 different iterations.  Found
a solution that moves 19980 regions; Going from a computed cost of 107.0 to a new cost of
8.8648
{code}
As you can see, though RegionReplicaHostCostFunction has high weight, its cost was 0. The
balancer was able to get to better overall cost by ignoring the weight from RegionReplicaHostCostFunction.

> StochasticLoadBalancer may incorrectly skip balancing due to skewed multiplier sum
> ----------------------------------------------------------------------------------
>
>                 Key: HBASE-17565
>                 URL: https://issues.apache.org/jira/browse/HBASE-17565
>             Project: HBase
>          Issue Type: Bug
>            Reporter: Ted Yu
>            Assignee: Ted Yu
>            Priority: Critical
>             Fix For: 2.0.0, 1.4.0
>
>         Attachments: 17565.v1.txt, 17565.v2.txt
>
>
> I was investigating why a 6 node cluster kept skipping balancing requests.
> Here were the region counts on the servers:
> 449, 448, 447, 449, 453, 0
> {code}
> 2017-01-26 22:04:47,145 INFO  [RpcServer.deafult.FPBQ.Fifo.handler=1,queue=0,port=16000]
balancer.StochasticLoadBalancer: Skipping load balancing because balanced cluster; total cost
is 127.0171157050385, sum multiplier is 111087.0 min cost which need balance is 0.05
> {code}
> The big multiplier sum caught my eyes. Here was what additional debug logging showed:
> {code}
> 2017-01-27 23:25:31,749 DEBUG [RpcServer.deafult.FPBQ.Fifo.handler=9,queue=0,port=16000]
balancer.StochasticLoadBalancer: class org.apache.hadoop.hbase.master.balancer.          StochasticLoadBalancer$RegionReplicaHostCostFunction
with multiplier 100000.0
> 2017-01-27 23:25:31,749 DEBUG [RpcServer.deafult.FPBQ.Fifo.handler=9,queue=0,port=16000]
balancer.StochasticLoadBalancer: class org.apache.hadoop.hbase.master.balancer.          StochasticLoadBalancer$RegionReplicaRackCostFunction
with multiplier 10000.0
> {code}
> Note however, that no table in the cluster used read replica.
> I can think of two ways of fixing this situation:
> 1. If there is no read replica in the cluster, ignore the multipliers for the above two
functions.
> 2. When cost() returned by the CostFunction is 0 (or very very close to 0.0), ignore
the multiplier.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

Mime
View raw message