hbase-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "ramkrishna.s.vasudevan (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HBASE-14921) Memory optimizations
Date Fri, 12 Aug 2016 05:36:22 GMT

    [ https://issues.apache.org/jira/browse/HBASE-14921?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15418386#comment-15418386
] 

ramkrishna.s.vasudevan commented on HBASE-14921:
------------------------------------------------

bq.1. Size arithmetic. I'll be publishing a design proposal for this issue here and on HBASE-16229
soon, let's manage the rest of the discussion there. There seems to be an elegant solution,
which I even have a partial code for. Can post a suggested patch immediately after HBASE-14921
is committed.
Yes. We are infact waiting for gettig in HBASE-16229 after this JIRA is done. Size arithmetic
is important for us for the offheap accounting also.

bq.The current patch circumvents the "too many open files" with a simple heuristic that prevents
too many segments from piling up. However, we realize that this issue might require more discussion,
so let's open a new jira for it. 
Makes sense. So were you able to observe the problems that we highlighted in the previous
discussions - We got two problems one is the invalid negative size for memstore and the 'Too
many open files'?
Were you able to get this patch running with the PE tool and the configs that we had updated
in the earlier discussions?

bq.And finally, the crown jewel - the CellChunkMap project. Its main value is providing platform
for the Off-Heap Memory project. Here too, let us create a new jira, focusing on the following:
Yes. Once you can get the CellChunkMap code into it - we can help you with the new type of
Cell to avoid chunk copying and the MSLAB chunk creation related code which we have maintained
internally.

bq.The Big Cells issue.
+1 to address this. Need to see what best can be done here. May be discuss what we call as
a BigCell. 

> Memory optimizations
> --------------------
>
>                 Key: HBASE-14921
>                 URL: https://issues.apache.org/jira/browse/HBASE-14921
>             Project: HBase
>          Issue Type: Sub-task
>    Affects Versions: 2.0.0
>            Reporter: Eshcar Hillel
>            Assignee: Anastasia Braginsky
>         Attachments: CellBlocksSegmentInMemStore.pdf, CellBlocksSegmentinthecontextofMemStore(1).pdf,
HBASE-14921-V01.patch, HBASE-14921-V02.patch, HBASE-14921-V03.patch, HBASE-14921-V04-CA-V02.patch,
HBASE-14921-V04-CA.patch, HBASE-14921-V05-CAO.patch, HBASE-14921-V06-CAO.patch, HBASE-14921-V08-CAO.patch,
HBASE-14921-V09-CAO.patch, InitialCellArrayMapEvaluation.pdf, IntroductiontoNewFlatandCompactMemStore.pdf,
MemStoreSizes.pdf, MemstoreItrCountissue.patch, NewCompactingMemStoreFlow.pptx
>
>
> Memory optimizations including compressed format representation and offheap allocations



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message