Mailing-List: contact issues-help@hbase.apache.org; run by ezmlm
Precedence: bulk
Date: Thu, 26 Jan 2017 17:28:24 +0000 (UTC)
From: "stack (JIRA)" <jira@apache.org>
To: issues@hbase.apache.org
Message-ID: <JIRA.13037134.1485196921000.4806.1485451704857@Atlassian.JIRA>
In-Reply-To: <JIRA.13037134.1485196921000@Atlassian.JIRA>
References: <JIRA.13037134.1485196921000@Atlassian.JIRA> <JIRA.13037134.1485196921924@jira-lw-us.apache.org>
Subject: [jira] [Commented] (HBASE-17515) Reduce memory footprint of
 RegionLoads kept by StochasticLoadBalancer
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
archived-at: Thu, 26 Jan 2017 17:28:34 -0000


    [ https://issues.apache.org/jira/browse/HBASE-17515?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15840097#comment-15840097 ] 

stack commented on HBASE-17515:
-------------------------------

I try to follow your reasoning:

+ You leave out class overhead. And aren't you supposed to multiple by 15, the number of regions to keep around? (Thats what I did). Regardless, you don't answer the question which is is 24MB (or 24MB * 15) too much data to  keep around for sake of a single metric?
+ You make some soft argument about 'good practice' cutting down memory in general but no work to show how this here is even a problem area in master memory management.
+ I ask if rather than do our own accounting, we should not defer to package and you turn the question around to have me do the work (Since you ask, Data Sketches, a stochastic streaming algorithms library looks promising especially as we need to start using it throughout)
+ You then pervert my general shaving of RL size question into a query about the four fields extracted here.

Please provide our gracious contributors with better quality input/direction.


> Reduce memory footprint of RegionLoads kept by StochasticLoadBalancer
> ---------------------------------------------------------------------
>
>                 Key: HBASE-17515
>                 URL: https://issues.apache.org/jira/browse/HBASE-17515
>             Project: HBase
>          Issue Type: Improvement
>            Reporter: Ted Yu
>            Assignee: Tim Brown
>             Fix For: 2.0.0, 1.4.0
>
>         Attachments: 17515.branch-1.v2.txt, 17515.v1.txt, 17515.v2.txt
>
>
> Currently StochasticLoadBalancer uses the following fields of RegionLoad :
> {code}
>       return rl.getReadRequestsCount();
>       return rl.getWriteRequestsCount();
>       return rl.getMemStoreSizeMB();
>       return rl.getStorefileSizeMB();
> {code}
> However, RegionLoad refers to ClusterStatusProtos.RegionLoad which has 18 fields.
> This means we keep hbase.master.balancer.stochastic.numRegionLoadsToRemember (default value 15) RegionLoad's in memory but only use a small portion of them.
> This JIRA is to consider creating POJO which wraps the above 4 fields so that the memory footprint can be lowered.
> Lowering memory footprint would allow wider sliding window to be configured for load balancer while limiting the impact to heap consumption.


--
This message was sent by Atlassian JIRA
(v6.3.4#6332)