hbase-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Yanping Wang (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HBASE-14613) Remove MemStoreChunkPool?
Date Sat, 26 Mar 2016 05:27:25 GMT

    [ https://issues.apache.org/jira/browse/HBASE-14613?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15212814#comment-15212814

Yanping Wang commented on HBASE-14613:

Hi, Elliott

>>Setting it lower doesn't help since this is all about the promotion from young to
old an g1 preferring not to do mixed. But we tried it.

No, no, not set to lower, instead, calculate the size of pinned in Old and set this number
for example, if you have 100GB heap, you calculate those "pinned old forever" are about 30GB,
then most likely the default 45% (45GB) will cause trouble for you. if after Mixed GC completes,
the heap  Occupancy is above 45GB, concurrent marking cycle will start immediately to prepare
next series of Mixed GC, which gives you the symptom of  "get pauses even though there is
a lot of memory left." 
so if for 100GB heap and 30GB pinned Old, you might have to increase -XX:InitiatingHeapOccupancyPercent
 to 60 or 65. so you are able to use more heap memory.  you can find those info by using -XX:+PrintAdaptiveSizePolicy

>>Setting this higher helped a lot but we have enough happening on these boxes that
it started to hurt performance.

after you let more heap to be used, the total number of Mixed GC is likely will be reduced,
which could cause full GC if we are not careful. so increase ConcGCThread helps concurrent
marking completes faster. just consider which one will hurt performance more? (1) take risk
of Full GC (2) 80 Mixed GCs using 4 concurrent threads to do 10 concurrent marking each last
12 seconds (3) 40 Mixed GCs with 12 concurrent threads to do 5 concurrent marking each last
4 seconds.

>> Setting this lower just moved what caused the long gc's

The most expensive steps in GC are scrub the remember set and COPY live objects from one place
to another. after each concurrent marking cycle completes, by default G1 will try to do 8
mixed GC, first 1-5 might be easy for G1 so takes less time to complete, but the deeper it
goes, the harder it gets, the longer it takes. so I'd recommend HBase don't need to go that
deep. The reason is other than pinned forever objects, most HBase objects are short lived.
if this group of mixed GC find too hard to clean and copy some regions, maybe wait a while,
in next cycle of mixed GC, some objects will die and makes their regions easy to clean up
(no need copy any more). 

> Remove MemStoreChunkPool?
> -------------------------
>                 Key: HBASE-14613
>                 URL: https://issues.apache.org/jira/browse/HBASE-14613
>             Project: HBase
>          Issue Type: Brainstorming
>            Reporter: Lars Hofhansl
>            Priority: Minor
>         Attachments: 14613-0.98.txt, gc.png, writes.png
> I just stumbled across MemStoreChunkPool. The idea behind is to reuse chunks of allocations
rather than letting the GC handle this.
> Now, it's off by default, and it seems to me to be of dubious value. I'd recommend just
removing it.

This message was sent by Atlassian JIRA

View raw message