Mailing-List: contact issues-help@hbase.apache.org; run by ezmlm
Precedence: bulk
Date: Sat, 4 Feb 2017 04:30:52 +0000 (UTC)
From: "Ted Yu (JIRA)" <jira@apache.org>
To: issues@hbase.apache.org
Message-ID: <JIRA.13038591.1485562638000.32499.1486182652552@Atlassian.JIRA>
In-Reply-To: <JIRA.13038591.1485562638000@Atlassian.JIRA>
References: <JIRA.13038591.1485562638000@Atlassian.JIRA> <JIRA.13038591.1485562638556@jira-lw-us.apache.org>
Subject: [jira] [Commented] (HBASE-17565) StochasticLoadBalancer may
 incorrectly skip balancing due to skewed multiplier sum
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
archived-at: Sat, 04 Feb 2017 04:30:58 -0000


    [ https://issues.apache.org/jira/browse/HBASE-17565?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15852581#comment-15852581 ] 

Ted Yu commented on HBASE-17565:
--------------------------------

bq. This only effect when there are a very very big multiplier......

Big multiplier does exist. We need to consider the aggregate effect.

bq. Why we need a so big default multiplier for read replica?

The background is that when any two replicas among primary, secondary and tertiary replicas are on the same server, we lose benefit of read replica when this server goes down. See the javadoc for RegionReplicaHostCostFunction :
{code}
   * A cost function for region replicas. We give a very high cost to hosting
   * replicas of the same region in the same host. We do not prevent the case
   * though, since if numReplicas > numRegionServers, we still want to keep the
   * replica open.
{code}

> StochasticLoadBalancer may incorrectly skip balancing due to skewed multiplier sum
> ----------------------------------------------------------------------------------
>
>                 Key: HBASE-17565
>                 URL: https://issues.apache.org/jira/browse/HBASE-17565
>             Project: HBase
>          Issue Type: Bug
>            Reporter: Ted Yu
>            Assignee: Ted Yu
>            Priority: Critical
>             Fix For: 2.0.0, 1.4.0
>
>         Attachments: 17565.v1.txt, 17565.v2.txt, 17565.v3.txt
>
>
> I was investigating why a 6 node cluster kept skipping balancing requests.
> Here were the region counts on the servers:
> 449, 448, 447, 449, 453, 0
> {code}
> 2017-01-26 22:04:47,145 INFO  [RpcServer.deafult.FPBQ.Fifo.handler=1,queue=0,port=16000] balancer.StochasticLoadBalancer: Skipping load balancing because balanced cluster; total cost is 127.0171157050385, sum multiplier is 111087.0 min cost which need balance is 0.05
> {code}
> The big multiplier sum caught my eyes. Here was what additional debug logging showed:
> {code}
> 2017-01-27 23:25:31,749 DEBUG [RpcServer.deafult.FPBQ.Fifo.handler=9,queue=0,port=16000] balancer.StochasticLoadBalancer: class org.apache.hadoop.hbase.master.balancer.          StochasticLoadBalancer$RegionReplicaHostCostFunction with multiplier 100000.0
> 2017-01-27 23:25:31,749 DEBUG [RpcServer.deafult.FPBQ.Fifo.handler=9,queue=0,port=16000] balancer.StochasticLoadBalancer: class org.apache.hadoop.hbase.master.balancer.          StochasticLoadBalancer$RegionReplicaRackCostFunction with multiplier 10000.0
> {code}
> Note however, that no table in the cluster used read replica.
> I can think of two ways of fixing this situation:
> 1. If there is no read replica in the cluster, ignore the multipliers for the above two functions.
> 2. When cost() returned by the CostFunction is 0 (or very very close to 0.0), ignore the multiplier.


--
This message was sent by Atlassian JIRA
(v6.3.15#6346)