hbase-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Vladimir Rodionov (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (HBASE-14951) Make hbase.regionserver.maxlogs obsolete
Date Mon, 14 Dec 2015 19:25:46 GMT

     [ https://issues.apache.org/jira/browse/HBASE-14951?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Vladimir Rodionov updated HBASE-14951:
--------------------------------------
    Release Note: 
Rolling WAL events across a cluster can be highly correlated, hence flushing memstores, hence
triggering minor compactions, that can be promoted to major ones. These events are highly
correlated in time if there is a balanced write-load on the regions in a table. Default value
for maximum WAL files (* hbase.regionserver.maxlogs*), which controls WAL rolling events -
32 is too small for many modern deployments. 
Now we calculate this value dynamically (if not defined by user), using the following formula:

maxLogs = Math.max( 32, HBASE_HEAP_SIZE * memstoreRatio * 2/ LogRollSize), where

memstoreRatio is *hbase.regionserver.global.memstore.size*
LogRollSize is maximum WAL file size (default 0.95 * HDFS block size)

The following table gives the new default maximum log files values for several different Region
Server heap sizes:
||heap||memstore ratio||max logs||
|1G|40%|32|
|2G|40%|32|
|10G|40%|80|
|20G|40%|160|
|32G|40%|256|   
  

  was:
Rolling WAL events across a cluster can be highly correlated, hence flushing memstores, hence
triggering minor compactions, that can be promoted to major ones. These events are highly
correlated in time if there is a balanced write-load on the regions in a table. Default value
for maximum WAL files (* hbase.regionserver.maxlogs*), which controls WAL rolling events -
32 is too small for many modern deployments. 
Now we calculate this value dynamically (if not defined by user) using the following formula:

maxLogs = HBASE_HEAP_SIZE * memstoreRatio * 2/ LogRollSize, where

memstoreRatio is 
  


> Make hbase.regionserver.maxlogs obsolete
> ----------------------------------------
>
>                 Key: HBASE-14951
>                 URL: https://issues.apache.org/jira/browse/HBASE-14951
>             Project: HBase
>          Issue Type: Improvement
>            Reporter: Vladimir Rodionov
>            Assignee: Vladimir Rodionov
>            Priority: Minor
>             Fix For: 2.0.0
>
>         Attachments: HBASE-14951-v1.patch, HBASE-14951-v2.patch
>
>
> There was a discussion in HBASE-14388 related to maximum number of log files. It was
an agreement that we should calculate this number in a code but still need to honor user's
setting. 
> Maximum number of log files now is calculated as following:
>  maxLogs = HEAP_SIZE * memstoreRatio * 2/ LogRollSize



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message