Mailing-List: contact issues-help@hbase.apache.org; run by ezmlm
Precedence: bulk
Date: Thu, 28 Jan 2016 09:47:39 +0000 (UTC)
From: "Anoop Sam John (JIRA)" <jira@apache.org>
To: issues@hbase.apache.org
Message-ID: <JIRA.12917935.1449133900000.235168.1453974459905@Atlassian.JIRA>
In-Reply-To: <JIRA.12917935.1449133900000@Atlassian.JIRA>
References: <JIRA.12917935.1449133900000@Atlassian.JIRA>
 <JIRA.12917935.1449133900913@arcas>
Subject: [jira] [Commented] (HBASE-14918) In-Memory MemStore Flush and
 Compaction
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit


    [ https://issues.apache.org/jira/browse/HBASE-14918?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15121121#comment-15121121 ] 

Anoop Sam John commented on HBASE-14918:
----------------------------------------

As per the present trunk code, we can move it out.. I have done also..  But am not sure whether we need it inside the new Memstore impl, (with internal flush to pipeline and flush as CellBlock).. So I did not raise a Jira.

Why I say to move out is this work of copying the Cell data into a MSLAB area is not a Memstore impl detail.  Whatever be the Memstore impl (current or new) we need this.  Also I have done a patch for avoiding garbage what we create in write path (See HBASE015180) when MSLAB is on.  That is why I thought to make it an upper layer work than at the Memstore impl.   
I need to see how my patch can satisfy the need of new memstore impls

> In-Memory MemStore Flush and Compaction
> ---------------------------------------
>
>                 Key: HBASE-14918
>                 URL: https://issues.apache.org/jira/browse/HBASE-14918
>             Project: HBase
>          Issue Type: Umbrella
>    Affects Versions: 2.0.0
>            Reporter: Eshcar Hillel
>            Assignee: Eshcar Hillel
>             Fix For: 0.98.18
>
>         Attachments: CellBlocksSegmentDesign.pdf
>
>
> A memstore serves as the in-memory component of a store unit, absorbing all updates to the store. From time to time these updates are flushed to a file on disk, where they are compacted (by eliminating redundancies) and compressed (i.e., written in a compressed format to reduce their storage size).
> We aim to speed up data access, and therefore suggest to apply in-memory memstore flush. That is to flush the active in-memory segment into an intermediate buffer where it can be accessed by the application. Data in the buffer is subject to compaction and can be stored in any format that allows it to take up smaller space in RAM. The less space the buffer consumes the longer it can reside in memory before data is flushed to disk, resulting in better performance.
> Specifically, the optimization is beneficial for workloads with medium-to-high key churn which incur many redundant cells, like persistent messaging. 
> We suggest to structure the solution as 4 subtasks (respectively, patches). 
> (1) Infrastructure - refactoring of the MemStore hierarchy, introducing segment (StoreSegment) as first-class citizen, and decoupling memstore scanner from the memstore implementation;
> (2) Adding StoreServices facility at the region level to allow memstores update region counters and access region level synchronization mechanism;
> (3) Implementation of a new memstore (CompactingMemstore) with non-optimized immutable segment representation, and 
> (4) Memory optimization including compressed format representation and off heap allocations.
> This Jira continues the discussion in HBASE-13408.
> Design documents, evaluation results and previous patches can be found in HBASE-13408. 


--
This message was sent by Atlassian JIRA
(v6.3.4#6332)