hbase-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "stack (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (HBASE-14613) Remove MemStoreChunkPool?
Date Wed, 21 Oct 2015 22:39:27 GMT

     [ https://issues.apache.org/jira/browse/HBASE-14613?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

stack updated HBASE-14613:
--------------------------
    Attachment: writes.png
                gc.png

Here are some diagrams. 25 clients YCSB writing a single regionserver. I ran four test runs.
The first and last were WITHOUT memstorechunkpool.  The second and third were WITH memstorechunkpool
enabled. I set it to be .25 the size of memstore and I set initial size to 0.

The diagrams are GC and write rates (the fourth write run is missing from the diagram...)

So, high-level, write rates are same with or without patch. No surprise.

The CMS GCs are less when the pool is in place. No surprise given we are not recreating chunks
but reusing them.

What is surprising is that the new gen seems to be doing more collections. I'd have to dig
in why.... (Guess would be that this pool is throwing off the ergonomic sizing of new vs old...
would have to check).

The pool prints stats and the stats look nice:

{code}
2015-10-21 11:04:07,026 DEBUG [StoreOpener-590f2ef9a43c9fe507f0f1fb415ca50e-1-MemStoreChunkPool
Statistics] regionserver.MemStoreChunkPool: Stats: current pool size=9,created chunk count=39,reused
chunk count=3170,reuseRatio=98.78%
2015-10-21 11:09:07,026 DEBUG [StoreOpener-590f2ef9a43c9fe507f0f1fb415ca50e-1-MemStoreChunkPool
Statistics] regionserver.MemStoreChunkPool: Stats: current pool size=8,created chunk count=39,reused
chunk count=3322,reuseRatio=98.84%
2015-10-21 11:14:07,026 DEBUG [StoreOpener-590f2ef9a43c9fe507f0f1fb415ca50e-1-MemStoreChunkPool
Statistics] regionserver.MemStoreChunkPool: Stats: current pool size=14,created chunk count=39,reused
chunk count=3584,reuseRatio=98.92%
2015-10-21 11:19:07,026 DEBUG [StoreOpener-590f2ef9a43c9fe507f0f1fb415ca50e-1-MemStoreChunkPool
Statistics] regionserver.MemStoreChunkPool: Stats: current pool size=30,created chunk count=39,reused
chunk count=3829,reuseRatio=98.99%
2015-10-21 11:24:07,026 DEBUG [StoreOpener-590f2ef9a43c9fe507f0f1fb415ca50e-1-MemStoreChunkPool
Statistics] regionserver.MemStoreChunkPool: Stats: current pool size=37,created chunk count=39,reused
chunk count=4092,reuseRatio=99.06%
2015-10-21 11:29:07,026 DEBUG [StoreOpener-590f2ef9a43c9fe507f0f1fb415ca50e-1-MemStoreChunkPool
Statistics] regionserver.MemStoreChunkPool: Stats: current pool size=12,created chunk count=39,reused
chunk count=4352,reuseRatio=99.11%
2015-10-21 11:34:07,026 DEBUG [StoreOpener-590f2ef9a43c9fe507f0f1fb415ca50e-1-MemStoreChunkPool
Statistics] regionserver.MemStoreChunkPool: Stats: current pool size=14,created chunk count=39,reused
chunk count=4609,reuseRatio=99.16%
2015-10-21 11:39:07,026 DEBUG [StoreOpener-590f2ef9a43c9fe507f0f1fb415ca50e-1-MemStoreChunkPool
Statistics] regionserver.MemStoreChunkPool: Stats: current pool size=20,created chunk count=39,reused
chunk count=4865,reuseRatio=99.20%
2015-10-21 11:44:07,026 DEBUG [StoreOpener-590f2ef9a43c9fe507f0f1fb415ca50e-1-MemStoreChunkPool
Statistics] regionserver.MemStoreChunkPool: Stats: current pool size=17,created chunk count=39,reused
chunk count=5128,reuseRatio=99.25%
2015-10-21 11:49:07,026 DEBUG [StoreOpener-590f2ef9a43c9fe507f0f1fb415ca50e-1-MemStoreChunkPool
Statistics] regionserver.MemStoreChunkPool: Stats: current pool size=20,created chunk count=39,reused
chunk count=5377,reuseRatio=99.28%
2015-10-21 11:54:07,026 DEBUG [StoreOpener-590f2ef9a43c9fe507f0f1fb415ca50e-1-MemStoreChunkPool
Statistics] regionserver.MemStoreChunkPool: Stats: current pool size=17,created chunk count=39,reused
chunk count=5630,reuseRatio=99.31%
2015-10-21 11:59:07,026 DEBUG [StoreOpener-590f2ef9a43c9fe507f0f1fb415ca50e-1-MemStoreChunkPool
Statistics] regionserver.MemStoreChunkPool: Stats: current pool size=23,created chunk count=39,reused
chunk count=5871,reuseRatio=99.34%
2015-10-21 12:04:07,026 DEBUG [StoreOpener-590f2ef9a43c9fe507f0f1fb415ca50e-1-MemStoreChunkPool
Statistics] regionserver.MemStoreChunkPool: Stats: current pool size=22,created chunk count=39,reused
chunk count=6122,reuseRatio=99.37%
2015-10-21 12:09:07,026 DEBUG [StoreOpener-590f2ef9a43c9fe507f0f1fb415ca50e-1-MemStoreChunkPool
Statistics] regionserver.MemStoreChunkPool: Stats: current pool size=36,created chunk count=39,reused
chunk count=6216,reuseRatio=99.38%
2015-10-21 12:14:07,026 DEBUG [StoreOpener-590f2ef9a43c9fe507f0f1fb415ca50e-1-MemStoreChunkPool
Statistics] regionserver.MemStoreChunkPool: Stats: current pool size=36,created chunk count=39,reused
chunk count=6216,reuseRatio=99.38%
2015-10-21 12:19:07,026 DEBUG [StoreOpener-590f2ef9a43c9fe507f0f1fb415ca50e-1-MemStoreChunkPool
Statistics] regionserver.MemStoreChunkPool: Stats: current pool size=36,created chunk count=39,reused
chunk count=6216,reuseRatio=99.38%
{code}

The reuse is impressive.

So, the pool could be good. I'd change it so the stats ran out of an executor rather than
as a dedicated thread as a Chore. We'd have to figure the sizing too. The initial sizing can
be zero. Thats fine.  Let it ramp up even if it means chunks have to traverse survivor space
a few times.... (would be sweet if could allocated direct on old gen but that seems hard to
do reliably -- could spend some more time digging in on this).   What for the max size?  Currently
it is a proportion of memstore size.  Could study running machine to see how many chunks are
outstanding when a write load going on.  Could then set the pool to be this size.  But, we
need to have a pool that can grow and shrink.  There is the ergonomic resize of the memstore
that will resize memstore if we are doing many more writes than reads. The pool should adjust
accordingly. The pool should also adjust if it has not been accessed in a long time. We do
not want the case where we have all memstores holding on to big pools because we just underwent
a furious write load but now it has abated... 

> Remove MemStoreChunkPool?
> -------------------------
>
>                 Key: HBASE-14613
>                 URL: https://issues.apache.org/jira/browse/HBASE-14613
>             Project: HBase
>          Issue Type: Brainstorming
>            Reporter: Lars Hofhansl
>            Priority: Minor
>         Attachments: 14613-0.98.txt, gc.png, writes.png
>
>
> I just stumbled across MemStoreChunkPool. The idea behind is to reuse chunks of allocations
rather than letting the GC handle this.
> Now, it's off by default, and it seems to me to be of dubious value. I'd recommend just
removing it.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message