hbase-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Enis Soztutar (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HBASE-14215) Default cost used for PrimaryRegionCountSkewCostFunction is not sufficient
Date Tue, 01 Sep 2015 23:15:46 GMT

    [ https://issues.apache.org/jira/browse/HBASE-14215?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14726404#comment-14726404
] 

Enis Soztutar commented on HBASE-14215:
---------------------------------------

Thanks Biju for coming up with these numbers. Your summary above is pretty good. It seems
pretty possible that we cannot go out of the local minimas in the search for the "global optimum"
when we are doing greedy descent search in the SLB when we try with 3 racks and 9 servers.
Possible that high costs like 4000 helps overcome the local minima coming from the candidate
temporary plans that co-locates the region replicas in the same rack or cluster (those with
10K cost). Since 3 racks and region replication = 3, LB cannot "simulate" new candidate plans
effectively.  

The rack cost is needed for cases where losing a rack should still not lose a replica (by
ensuring that we are not co-locating in the same rack), but it is optional. Would it be possible
to repeat the experiment with {{hbase.master.balancer.stochastic.regionReplicaRackCostKey=0}}
which should effectively disable the rack based cost. The expectation is that with rack based
costs gone, the LB would be able to come up with better plans even with {{hbase.master.balancer.stochastic.primaryRegionCountCost=500}}.


> Default cost used for PrimaryRegionCountSkewCostFunction is not sufficient 
> ---------------------------------------------------------------------------
>
>                 Key: HBASE-14215
>                 URL: https://issues.apache.org/jira/browse/HBASE-14215
>             Project: HBase
>          Issue Type: Bug
>          Components: Balancer
>            Reporter: Biju Nair
>            Priority: Minor
>         Attachments: 14215-v1.txt
>
>
> Current multiplier of 500 used in the stochastic balancer cost function {{PrimaryRegionCountSkewCostFunction}}
to calculate the cost of  total primary replication skew doesn't seem to be sufficient to
prevent the skews (Refer HBASE-14110). We would want the default cost to be a higher value
so that skews in primary region replica has higher cost. The following is the test result
by setting the multiplier value to 10000 (same as the region replica rack cost multiplier)
on a 3 Rack 9 RS node cluster which seems to get the balancer distribute the primaries uniformly.
> *Initial Primary replica distribution - using the current multiplier*	
>  |r1n10|  102|
>  |r1n11|  85|
>  |r1n9|    88|
>  |r2n10|  120|
>  |r2n11|  120|
>  |r2n9|   124|
>  |r3n10|  135|
>  |r3n11|  124|
>  |r3n9|    129|
> *After long duration of read & writes - using current multiplier*	
> | r1n10|  102|
> | r1n11|  85|
> | r1n9|    88|
> | r2n10|  120|
> | r2n11|  120|
> | r2n9 |   124|
> | r3n10|  135|
> | r3n11|  124|
> | r3n9|    129|
> *After manual balancing* 	
> | r1n10|  102|
> | r1n11|  85|
> | r1n9|    88|
> | r2n10|  120|
> | r2n11|  120|
> | r2n9 |   124|
> | r3n10|  135|
> | r3n11|  124|
> | r3n9|    129|
> *Increased multiplier for primaryRegionCountSkewCost to 10000*	
> | r1n10|  114|
> | r1n11 | 113|
> | r1n9 |   114|
> | r2n10|  114|
> | r2n11|  114|
> | r2n9 |   113|
> | r3n10|  115|
> | r3n11|  115|
> | r3n9 |   115 |
> Setting the {{PrimaryRegionCountSkewCostFunction}} multiplier value to 10000 should help
HBase general use.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message