hbase-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Ted Yu (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HBASE-17515) Reduce memory footprint of RegionLoads kept by StochasticLoadBalancer
Date Thu, 26 Jan 2017 17:56:24 GMT

    [ https://issues.apache.org/jira/browse/HBASE-17515?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15840131#comment-15840131
] 

Ted Yu commented on HBASE-17515:
--------------------------------

bq. aren't you supposed to multiple by 15, the number of regions to keep around

Clarification about the 15 (default value for hbase.master.balancer.stochastic.numRegionLoadsToRemember).
See the code in StochasticLoadBalancer#updateRegionLoad() :
{code}
      for (Entry<byte[], RegionLoad> entry : sl.getRegionsLoad().entrySet()) {
        Deque<BalancerRegionLoad> rLoads = oldLoads.get(Bytes.toString(entry.getKey()));
        if (rLoads == null) {
          // There was nothing there
          rLoads = new ArrayDeque<BalancerRegionLoad>();
        } else if (rLoads.size() >= numRegionLoadsToRemember) {
{code}
For each region, there is a Deque which keeps 15 RegionLoad's.
So 15 is per region, not the number of regions to keep around.

I am collecting size comparison data through Instrumentation. Will be back with more response.

> Reduce memory footprint of RegionLoads kept by StochasticLoadBalancer
> ---------------------------------------------------------------------
>
>                 Key: HBASE-17515
>                 URL: https://issues.apache.org/jira/browse/HBASE-17515
>             Project: HBase
>          Issue Type: Improvement
>            Reporter: Ted Yu
>            Assignee: Tim Brown
>             Fix For: 2.0.0, 1.4.0
>
>         Attachments: 17515.branch-1.v2.txt, 17515.v1.txt, 17515.v2.txt
>
>
> Currently StochasticLoadBalancer uses the following fields of RegionLoad :
> {code}
>       return rl.getReadRequestsCount();
>       return rl.getWriteRequestsCount();
>       return rl.getMemStoreSizeMB();
>       return rl.getStorefileSizeMB();
> {code}
> However, RegionLoad refers to ClusterStatusProtos.RegionLoad which has 18 fields.
> This means we keep hbase.master.balancer.stochastic.numRegionLoadsToRemember (default
value 15) RegionLoad's in memory but only use a small portion of them.
> This JIRA is to consider creating POJO which wraps the above 4 fields so that the memory
footprint can be lowered.
> Lowering memory footprint would allow wider sliding window to be configured for load
balancer while limiting the impact to heap consumption.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message