hbase-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Anastasia Braginsky (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HBASE-14921) Memory optimizations
Date Thu, 11 Aug 2016 12:35:20 GMT

    [ https://issues.apache.org/jira/browse/HBASE-14921?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15417138#comment-15417138
] 

Anastasia Braginsky commented on HBASE-14921:
---------------------------------------------

The current patch includes fixes to all accepted code review comments. Thank you [~ram_krish],
[~anoop.hbase], and [~stack] for your comments!!

The goal of this patch is finalize the implementation reflecting the design published on this
Jira. Our intent is to isolate non-critical issues to the following jira's, before the current
code becomes unmanageable :) The current patch will go thru the normal review cycle, of course.
In the future, we'll be shooting for much smaller jira's to converge faster.

Suggested roadmap of taking care about the related issues.
1. Size arithmetic. I'll be publishing a design proposal for this issue here and on HBASE-16229
soon, let's manage the rest of the discussion there.  There seems to be an elegant solution,
which I even have a partial code for. Can post a suggested patch immediately after HBASE-14921
is committed.
2. In-memory flattening/compaction policy. The current patch circumvents the "too many open
files" with a simple heuristic that prevents too many segments from piling up. However, we
realize that this issue might require more discussion, so let's open a new jira for it.  We
can deal with all performance issues and  configuration possibilities there.
3. And finally, the crown jewel - the CellChunkMap project. Its main value is providing platform
for the Off-Heap Memory project. Here too, let us create a new jira, focusing on the following:
    - The CellChunkMap implementation itself (already prototyped but not integrated yet)
    - Related design issues (some refactoring of MemStoreChunkPool, MSLAB and HeapMSLAB).
    - Flattening to CellChunkMap (integrating with new [~anoop.hbase] and [~ram_krish] code)
    - The Big Cells issue.
We might need to manage this project as an umbrella jira, to de-clutter the space :)

Looking forward to your review comments.
Thank you!

> Memory optimizations
> --------------------
>
>                 Key: HBASE-14921
>                 URL: https://issues.apache.org/jira/browse/HBASE-14921
>             Project: HBase
>          Issue Type: Sub-task
>    Affects Versions: 2.0.0
>            Reporter: Eshcar Hillel
>            Assignee: Anastasia Braginsky
>         Attachments: CellBlocksSegmentInMemStore.pdf, CellBlocksSegmentinthecontextofMemStore(1).pdf,
HBASE-14921-V01.patch, HBASE-14921-V02.patch, HBASE-14921-V03.patch, HBASE-14921-V04-CA-V02.patch,
HBASE-14921-V04-CA.patch, HBASE-14921-V05-CAO.patch, HBASE-14921-V06-CAO.patch, HBASE-14921-V08-CAO.patch,
HBASE-14921-V09-CAO.patch, InitialCellArrayMapEvaluation.pdf, IntroductiontoNewFlatandCompactMemStore.pdf,
MemstoreItrCountissue.patch, NewCompactingMemStoreFlow.pptx
>
>
> Memory optimizations including compressed format representation and offheap allocations



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message