hbase-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "ramkrishna.s.vasudevan (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HBASE-14921) Memory optimizations
Date Fri, 22 Jul 2016 06:52:20 GMT

    [ https://issues.apache.org/jira/browse/HBASE-14921?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15389043#comment-15389043

ramkrishna.s.vasudevan commented on HBASE-14921:

Did the testing with compacting memstore. 
Based on our write path offheaping work we had arrived at certain config params and gc tuning.
Onheap case : Xmx : 32  G
Offheap case Xmx : 12 G
Offheap memory for memstore : 32 G

global memstore size : 0.42
"-XX:+UseG1GC -XX:MaxGCPauseMillis=200 -XX:InitiatingHeapOccupancyPercent=60 -XX:G1HeapWastePercent=20

PE :  ./hbase org.apache.hadoop.hbase.PerformanceEvaluation --nomapred --presplit=50 --size=150
--columns=50 --valueSize=200 --writeToWAL=false --bloomFilter=NONE --inmemoryCompaction=false/true
 randomWrite 50

We were able to run and get results for the default memstore (onheap), default memstore(offheap),
Pipelinememstore (Internal impl where all segments are flushed with no in memory compaction)
- flatten to ARRAY_MAP and flatten to CHUNK_MAP.

But with compacting memstore we are not able to run and get results as with the offheap case
12G is just not enough because we have duplicates while doing in memory compaction. The segments
are read once again to create a compacted segment and it is creating lot of garbage and more
memory need.

With onheap case I increased the heap size to 50G (as with 32G I was not able to run) still
the compaction was taking lot of memory and most importantly the disk flushes and disk compactions
were getting piled up and finally got 'Too many open files' exception.

So I copied the latest trunk patch and gave a run and it had other exception due to invalid
region size settings as it was going negative. 
2016-07-22 16:14:51,066 ERROR [RpcServer.FifoWFPBQ.default.handler=60,queue=0,port=16041-inmemoryCompactions-1469184056544]
regionserver.HRegion: Asked to modify this region's (TestTable,00000000000000000005033136,1469184040624.26d47b5614da8885fd0768b66a5d6457.)
memstoreSize to a negative value which is incorrect. Current memstoreSize=33534312, delta=-33570288
        at org.apache.hadoop.hbase.regionserver.HRegion.addAndGetGlobalMemstoreSize(HRegion.java:1125)
        at org.apache.hadoop.hbase.regionserver.RegionServicesForStores.addAndGetGlobalMemstoreSize(RegionServicesForStores.java:72)
        at org.apache.hadoop.hbase.regionserver.CompactionPipeline.swap(CompactionPipeline.java:111)
        at org.apache.hadoop.hbase.regionserver.CompactingMemStore.swapCompactedSegments(CompactingMemStore.java:208)
        at org.apache.hadoop.hbase.regionserver.MemStoreCompactor.doCompaction(MemStoreCompactor.java:173)
        at org.apache.hadoop.hbase.regionserver.MemStoreCompactor.start(MemStoreCompactor.java:114)
        at org.apache.hadoop.hbase.regionserver.CompactingMemStore.flushInMemory(CompactingMemStore.java:311)
        at org.apache.hadoop.hbase.regionserver.CompactingMemStore$InMemoryFlushRunnable.run(CompactingMemStore.java:384)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)

So in order to avoid compaction I just ensure that only flattening happens (because in the
default case that is what you are trying to do). Even here the problem is that since only
the flush happens on the tail part we keep pressurizing the memstore and we have lot of blocked
regions. Also the tail flush is creating lot of small files and this is leading to more compactions
and hence  again getting 'Too  many file open' exception. 

When i did the above change to the offheap memstore (allow only flattening) case the problem
is similar but here the tests are taking longer time to complete as every time only a small
portion of the memstore is getting flushed out. 

So we feel that if we go only with flattening then we should ensure that entire segments in
pipeline has to be flushed.

> Memory optimizations
> --------------------
>                 Key: HBASE-14921
>                 URL: https://issues.apache.org/jira/browse/HBASE-14921
>             Project: HBase
>          Issue Type: Sub-task
>    Affects Versions: 2.0.0
>            Reporter: Eshcar Hillel
>            Assignee: Anastasia Braginsky
>         Attachments: CellBlocksSegmentInMemStore.pdf, CellBlocksSegmentinthecontextofMemStore(1).pdf,
HBASE-14921-V01.patch, HBASE-14921-V02.patch, HBASE-14921-V03.patch, HBASE-14921-V04-CA-V02.patch,
HBASE-14921-V04-CA.patch, HBASE-14921-V05-CAO.patch, HBASE-14921-V06-CAO.patch, InitialCellArrayMapEvaluation.pdf,
> Memory optimizations including compressed format representation and offheap allocations

This message was sent by Atlassian JIRA

View raw message