hbase-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "ramkrishna.s.vasudevan (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HBASE-14921) Memory optimizations
Date Wed, 17 Aug 2016 10:03:20 GMT

    [ https://issues.apache.org/jira/browse/HBASE-14921?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15424241#comment-15424241

ramkrishna.s.vasudevan commented on HBASE-14921:

I did a test with the latest patch - V10 version.

With 40G heap space and the below PE configs,
 ./hbase org.apache.hadoop.hbase.PerformanceEvaluation --nomapred --presplit=50 --size=150
--columns=50 --valueSize=200 --writeToWAL=false --bloomFilter=NONE --inmemoryCompaction=false/true
 randomWrite 50
 Am still not able to run with InMemoryCompaction enabled and we get a FullGC. With default
memstore am able to run.
Good news is that in that 2 min that the sytem ran there was no negative flush size happening.
So it may be we are good here.

The reason should be because, 
>From the code I think though flattening is enabled, we see if the number of segments is
> 3 and so we do compaction. I think it goes under that logic. Speculative scan is default
Since compaction is happening I think we have duplicated segments till the new one is updated
in the pipeline. This is what I was saying in the last test also. 
You said you were able to run with YCSB. With YCSB what were the number of cols and the data
As said I won't block the check in - we need to identify the reason for the Full GC and add
some policy or do the all segment flush thing that we had done based on the policy. Lets discuss
that in other JIRA.

> Memory optimizations
> --------------------
>                 Key: HBASE-14921
>                 URL: https://issues.apache.org/jira/browse/HBASE-14921
>             Project: HBase
>          Issue Type: Sub-task
>    Affects Versions: 2.0.0
>            Reporter: Eshcar Hillel
>            Assignee: Anastasia Braginsky
>         Attachments: CellBlocksSegmentInMemStore.pdf, CellBlocksSegmentinthecontextofMemStore(1).pdf,
HBASE-14921-V01.patch, HBASE-14921-V02.patch, HBASE-14921-V03.patch, HBASE-14921-V04-CA-V02.patch,
HBASE-14921-V04-CA.patch, HBASE-14921-V05-CAO.patch, HBASE-14921-V06-CAO.patch, HBASE-14921-V08-CAO.patch,
HBASE-14921-V09-CAO.patch, HBASE-14921-V10-CAO.patch, HBASE-14921-V11-CAO.patch, InitialCellArrayMapEvaluation.pdf,
IntroductiontoNewFlatandCompactMemStore.pdf, MemStoreSizes.pdf, MemstoreItrCountissue.patch,
> Memory optimizations including compressed format representation and offheap allocations

This message was sent by Atlassian JIRA

View raw message