hbase-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "stack (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HBASE-17515) Reduce memory footprint of RegionLoads kept by StochasticLoadBalancer
Date Thu, 26 Jan 2017 05:14:26 GMT

    [ https://issues.apache.org/jira/browse/HBASE-17515?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15839247#comment-15839247

stack commented on HBASE-17515:

Patch seems ok but there is no accounting of what the cost of keeping the current type RL
incurs. Without it, why would we introduce a new type? I'd think there'd be more justification
than just 18 fields vs 4?

What is 1M Regions * a class of 18 fields vs 4 fields (with class overhead)? Let's say one
is 1/4 the size of the former..... With 1M regions keeping 15 of them with RL being 256 bytes
say in size, thats about 4G. If we say that the new smaller class is 1/4, that's 1G. Is 4G
prohibitive on a master node w/ 256G of RAM? Is 1G given this is but one of many inputs to
the balancer?

Should we be looking to a stats package instead, one that economically keeps up our sliding
window accounting? Or should we be working on shaving the size of RL which would have benefit
all around?

> Reduce memory footprint of RegionLoads kept by StochasticLoadBalancer
> ---------------------------------------------------------------------
>                 Key: HBASE-17515
>                 URL: https://issues.apache.org/jira/browse/HBASE-17515
>             Project: HBase
>          Issue Type: Improvement
>            Reporter: Ted Yu
>            Assignee: Tim Brown
>             Fix For: 2.0.0, 1.4.0
>         Attachments: 17515.branch-1.v2.txt, 17515.v1.txt, 17515.v2.txt
> Currently StochasticLoadBalancer uses the following fields of RegionLoad :
> {code}
>       return rl.getReadRequestsCount();
>       return rl.getWriteRequestsCount();
>       return rl.getMemStoreSizeMB();
>       return rl.getStorefileSizeMB();
> {code}
> However, RegionLoad refers to ClusterStatusProtos.RegionLoad which has 18 fields.
> This means we keep hbase.master.balancer.stochastic.numRegionLoadsToRemember (default
value 15) RegionLoad's in memory but only use a small portion of them.
> This JIRA is to consider creating POJO which wraps the above 4 fields so that the memory
footprint can be lowered.

This message was sent by Atlassian JIRA

View raw message