hbase-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Eshcar Hillel (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HBASE-13408) HBase In-Memory Memstore Compaction
Date Tue, 14 Jul 2015 12:59:04 GMT

    [ https://issues.apache.org/jira/browse/HBASE-13408?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14626309#comment-14626309

Eshcar Hillel commented on HBASE-13408:

A comment and a request: we’ve yet to address the WAL truncating issue.
The problem is twofold:
(1) if the region comprises only an in-memory column (store) then flush may not occur for
a long time resulting in a big log which in turn may significantly increase MTTR. This is
(2) if the in-memory column (store) is part of a region with default stores then flushes do
occur, and the WAL truncates even entries it should not. Specifically it truncate entries
of the in-memory store that are still present in the memstore, that is, not eliminated by
compaction and not flushed to disk. This is a real threat to HBase durability guarantees.

The same solution can help avoid both problems. Currently the WAL uses a region counter to
mark the entries as well as to decide which entries are truncable. However, the memstore is
unaware of these sequence numbers and therefore cannot indicate which WAL entries should not
be truncated.
We would like to come up with a mechanism that allows the memstore and WAL to share the minimal
required information in order to ensure the data durability. We’d appreciate suggestions/insights.

> HBase In-Memory Memstore Compaction
> -----------------------------------
>                 Key: HBASE-13408
>                 URL: https://issues.apache.org/jira/browse/HBASE-13408
>             Project: HBase
>          Issue Type: New Feature
>            Reporter: Eshcar Hillel
>         Attachments: HBaseIn-MemoryMemstoreCompactionDesignDocument-ver02.pdf, HBaseIn-MemoryMemstoreCompactionDesignDocument.pdf,
> A store unit holds a column family in a region, where the memstore is its in-memory component.
The memstore absorbs all updates to the store; from time to time these updates are flushed
to a file on disk, where they are compacted. Unlike disk components, the memstore is not compacted
until it is written to the filesystem and optionally to block-cache. This may result in underutilization
of the memory due to duplicate entries per row, for example, when hot data is continuously
> Generally, the faster the data is accumulated in memory, more flushes are triggered,
the data sinks to disk more frequently, slowing down retrieval of data, even if very recent.
> In high-churn workloads, compacting the memstore can help maintain the data in memory,
and thereby speed up data retrieval. 
> We suggest a new compacted memstore with the following principles:
> 1.	The data is kept in memory for as long as possible
> 2.	Memstore data is either compacted or in process of being compacted 
> 3.	Allow a panic mode, which may interrupt an in-progress compaction and force a flush
of part of the memstore.
> We suggest applying this optimization only to in-memory column families.
> A design document is attached.
> This feature was previously discussed in HBASE-5311.

This message was sent by Atlassian JIRA

View raw message