hbase-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "stack (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HBASE-17515) Reduce memory footprint of RegionLoads kept by StochasticLoadBalancer
Date Thu, 26 Jan 2017 17:28:24 GMT

    [ https://issues.apache.org/jira/browse/HBASE-17515?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15840097#comment-15840097

stack commented on HBASE-17515:

I try to follow your reasoning:

+ You leave out class overhead. And aren't you supposed to multiple by 15, the number of regions
to keep around? (Thats what I did). Regardless, you don't answer the question which is is
24MB (or 24MB * 15) too much data to  keep around for sake of a single metric?
+ You make some soft argument about 'good practice' cutting down memory in general but no
work to show how this here is even a problem area in master memory management.
+ I ask if rather than do our own accounting, we should not defer to package and you turn
the question around to have me do the work (Since you ask, Data Sketches, a stochastic streaming
algorithms library looks promising especially as we need to start using it throughout)
+ You then pervert my general shaving of RL size question into a query about the four fields
extracted here.

Please provide our gracious contributors with better quality input/direction.

> Reduce memory footprint of RegionLoads kept by StochasticLoadBalancer
> ---------------------------------------------------------------------
>                 Key: HBASE-17515
>                 URL: https://issues.apache.org/jira/browse/HBASE-17515
>             Project: HBase
>          Issue Type: Improvement
>            Reporter: Ted Yu
>            Assignee: Tim Brown
>             Fix For: 2.0.0, 1.4.0
>         Attachments: 17515.branch-1.v2.txt, 17515.v1.txt, 17515.v2.txt
> Currently StochasticLoadBalancer uses the following fields of RegionLoad :
> {code}
>       return rl.getReadRequestsCount();
>       return rl.getWriteRequestsCount();
>       return rl.getMemStoreSizeMB();
>       return rl.getStorefileSizeMB();
> {code}
> However, RegionLoad refers to ClusterStatusProtos.RegionLoad which has 18 fields.
> This means we keep hbase.master.balancer.stochastic.numRegionLoadsToRemember (default
value 15) RegionLoad's in memory but only use a small portion of them.
> This JIRA is to consider creating POJO which wraps the above 4 fields so that the memory
footprint can be lowered.
> Lowering memory footprint would allow wider sliding window to be configured for load
balancer while limiting the impact to heap consumption.

This message was sent by Atlassian JIRA

View raw message