hbase-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Vladimir Rodionov (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (HBASE-14383) Compaction improvements
Date Wed, 09 Sep 2015 00:19:46 GMT

     [ https://issues.apache.org/jira/browse/HBASE-14383?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Vladimir Rodionov updated HBASE-14383:
--------------------------------------
    Description: 
Still major issue in many production environments. The general recommendation - disabling
region splitting and major compactions to reduce unpredictable IO/CPU spikes, especially during
peak times and running them manually during off peak times. Still do not resolve the issues
completely.

h3. Flush storms

* rolling WAL events across cluster can be highly correlated, hence flushing memstores, hence
triggering minor compactions, that can be promoted to major ones. These events are highly
correlated in time if there is a balanced write-load on the regions in a table.
*  the same is true for memstore flushing due to periodic memstore flusher operation. 

Both above may produce *flush storms* which are as bad as *compaction storms*. 

What can be done here. We can spread these events over time by randomizing (with jitter) several
 config options:
# hbase.regionserver.optionalcacheflushinterval
# hbase.regionserver.flush.per.changes
# hbase.regionserver.maxlogs   

h3. ExploringCompactionPolicy max compaction size
One more optimization can be added to ExploringCompactionPolicy. To limit size of a compaction
there is a config parameter one could use hbase.hstore.compaction.max.size. It would be nice
to have two separate limits: for peak and off peak hours.

h3. ExploringCompactionPolicy selection evaluation algorithm

Too simple? Selection with more files always wins, selection of smaller size wins if number
of files is the same. 

  was:
Still major issue in many production environments. The general recommendation - disabling
region splitting and major compactions to reduce unpredictable IO/CPU spikes, especially during
peak times and running them manually during off peak times. Still do not resolve the issues
completely.

h3. Flush storms

* rolling WAL events across cluster can be highly correlated, hence flushing memstores, hence
triggering minor compactions, that can be promoted to major ones. These events are highly
correlated in time if there is a balanced write-load on the regions in a table.
*  the same is true for memstore flushing due to periodic memstore flusher operation. 

Both above may produce *flush storms* which are as bad as *compaction storms*. 

What can be done here. We can spread these events over time by randomizing (with jitter) several
 config options:
# hbase.regionserver.optionalcacheflushinterval
# hbase.regionserver.flush.per.changes
# hbase.regionserver.maxlogs   

h3. ExploringCompactionPolicy max compaction size
One more optimization can be added to ExploringCompactionPolicy. To limit size of a compaction
there is a config parameter one could use hbase.hstore.compaction.max.size. It would be nice
to have two separate limits: for peak and off peak hours.

h3. ExploringCompactionPolicy selection evaluation algorithm

Just seems too simple: selection with more files always wins, selection of smaller size wins
if number of files is the same.


> Compaction improvements
> -----------------------
>
>                 Key: HBASE-14383
>                 URL: https://issues.apache.org/jira/browse/HBASE-14383
>             Project: HBase
>          Issue Type: Improvement
>            Reporter: Vladimir Rodionov
>            Assignee: Vladimir Rodionov
>             Fix For: 2.0.0
>
>
> Still major issue in many production environments. The general recommendation - disabling
region splitting and major compactions to reduce unpredictable IO/CPU spikes, especially during
peak times and running them manually during off peak times. Still do not resolve the issues
completely.
> h3. Flush storms
> * rolling WAL events across cluster can be highly correlated, hence flushing memstores,
hence triggering minor compactions, that can be promoted to major ones. These events are highly
correlated in time if there is a balanced write-load on the regions in a table.
> *  the same is true for memstore flushing due to periodic memstore flusher operation.

> Both above may produce *flush storms* which are as bad as *compaction storms*. 
> What can be done here. We can spread these events over time by randomizing (with jitter)
several  config options:
> # hbase.regionserver.optionalcacheflushinterval
> # hbase.regionserver.flush.per.changes
> # hbase.regionserver.maxlogs   
> h3. ExploringCompactionPolicy max compaction size
> One more optimization can be added to ExploringCompactionPolicy. To limit size of a compaction
there is a config parameter one could use hbase.hstore.compaction.max.size. It would be nice
to have two separate limits: for peak and off peak hours.
> h3. ExploringCompactionPolicy selection evaluation algorithm
> Too simple? Selection with more files always wins, selection of smaller size wins if
number of files is the same. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message