hbase-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Paul Tuckfield (JIRA)" <j...@apache.org>
Subject [jira] Commented: (HBASE-3327) For increment workloads, retain memstores in memory after flushing them
Date Wed, 15 Dec 2010 20:54:01 GMT

    [ https://issues.apache.org/jira/browse/HBASE-3327?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12971831#action_12971831

Paul Tuckfield commented on HBASE-3327:

I see the logic behind compact memory and cacheOnWrite, but still for some distribution of
keys being updated, the memory tradeoffs can favor memstore in terms of ram consumption. I
suppose the tradeoff point exists somewhere in reasonable tuning range. So it seems like this
gives the user control to understand their datalocality and make tuning tradeoffs.

If memstore reads are slower (presumably because of contention with writers to memstore) that
seems like a global problem, especially if check-and-miss is slow (I"m ignorant as to whether
checking existence of a key is as expensive as checking+readingvalue) Because that's the first
check any read must do, block cache, snapshot or physical IO, all check memstore first i think.

I'd very much like to test this just by a boolean setting allowing a snapshot to remain in
ram until the next memstore must be converted to a snapshot. I suspect 1 memstore plus one
snapshot gives most of the benefit, and is tuneable by existing memstore size affecting parameters.
But maybe this could be a memstore + N snapshots.

> For increment workloads, retain memstores in memory after flushing them
> -----------------------------------------------------------------------
>                 Key: HBASE-3327
>                 URL: https://issues.apache.org/jira/browse/HBASE-3327
>             Project: HBase
>          Issue Type: Improvement
>          Components: regionserver
>            Reporter: Karthik Ranganathan
> This is an improvement based on our observation of what happens in an increment workload.
The working set is typically small and is contained in the memstores. 
> 1. The reason the memstores get flushed is because the number of wal logs limit gets
> 2. This in turn triggers compactions, which evicts the block cache. 
> 3. Flushing of memstore and eviction of the block cache causes disk reads for increments
coming in after this because the data is no longer in memory.
> We could solve this elegantly by retaining the memstores AFTER they are flushed into
files. This would mean we can quickly populate the new memstore with the working set of data
from memory itself without having to hit disk. We can throttle the number of such memstores
we retain, or the memory allocated to it. In fact, allocating a percentage of the block cache
to this would give us a huge boost.

This message is automatically generated by JIRA.
You can reply to this email to add a comment to the issue online.

View raw message