hbase-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Ted Yu (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HBASE-17515) Reduce memory footprint of RegionLoads kept by StochasticLoadBalancer
Date Thu, 26 Jan 2017 11:40:24 GMT

    [ https://issues.apache.org/jira/browse/HBASE-17515?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15839598#comment-15839598

Ted Yu commented on HBASE-17515:

bq. justification than just 18 fields vs 4?

The 4 fields in BalancerRegionLoad would take 24 bytes (2 long's and 2 int's). For 1M Regions,
that would be ~24MB. If heap is 256G, this amount is pretty small.

Among the fields in ClusterStatusProtos.RegionLoad, there is storeCompleteSequenceId which
is of type List<org.apache.hadoop.hbase.shaded.protobuf.generated.ClusterStatusProtos.StoreSequenceId>.

The savings given by BalancerRegionLoad would better than 1/4.

bq. Is 4G prohibitive on a master node w/ 256G of RAM?

In hbase 2.0, more functionality is shifted to master (split / merge / some replication).
Preserving heap for those components would be good practice - considering potentially high
number of regions in deployment.

bq. Should we be looking to a stats package

Which stats package(s) is recommended ?

bq. should we be working on shaving the size of RL which would have benefit all around?

Let me dig into other classes which reference any of the 4 fields mentioned in the description
to see if they keep sliding window around.

> Reduce memory footprint of RegionLoads kept by StochasticLoadBalancer
> ---------------------------------------------------------------------
>                 Key: HBASE-17515
>                 URL: https://issues.apache.org/jira/browse/HBASE-17515
>             Project: HBase
>          Issue Type: Improvement
>            Reporter: Ted Yu
>            Assignee: Tim Brown
>             Fix For: 2.0.0, 1.4.0
>         Attachments: 17515.branch-1.v2.txt, 17515.v1.txt, 17515.v2.txt
> Currently StochasticLoadBalancer uses the following fields of RegionLoad :
> {code}
>       return rl.getReadRequestsCount();
>       return rl.getWriteRequestsCount();
>       return rl.getMemStoreSizeMB();
>       return rl.getStorefileSizeMB();
> {code}
> However, RegionLoad refers to ClusterStatusProtos.RegionLoad which has 18 fields.
> This means we keep hbase.master.balancer.stochastic.numRegionLoadsToRemember (default
value 15) RegionLoad's in memory but only use a small portion of them.
> This JIRA is to consider creating POJO which wraps the above 4 fields so that the memory
footprint can be lowered.

This message was sent by Atlassian JIRA

View raw message