hbase-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "stack (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (HBASE-13142) [PERF] Reuse the IPCUtil#buildCellBlock buffer
Date Fri, 06 Mar 2015 19:03:39 GMT

     [ https://issues.apache.org/jira/browse/HBASE-13142?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

stack updated HBASE-13142:
--------------------------
    Release Note: 
Adds buffer reuse sending Cell results. It is on by default and should not need configuration.
Improves GC profile and ups throughput. The benefit gets better the larger the row size returned.

The buffer reservoir is bounded at a maximum count after which we will start logging at WARN
level that the reservoir is running at capacity (returned buffers will be discarded and not
added back to the reservoir pool). Default maximum is twice the handler count: i.e. 2 * hbase.regionserver.handler.count.
This should be more than enough. Set the maximum with the new configuration: hbase.ipc.server.reservoir.max

The reservoir will not cache buffers in excess of hbase.ipc.server.reservoir.max.buffer.size
 The default is 10MB. This means that if a row is very large, then we will allocate a buffer
of the average size that is currently in the pool and we will then resize it till we can accommodate
the return. These resizes are expensive. The resultant buffer will be used and then discarded.

To check how the reservoir is doing, enable trace level logging for a few seconds on a regionserver.
You can do this from the regionserver UI. See 'Log Level'. Set org.apache.hadoop.hbase.io.BoundedByteBufferPool
to TRACE. The BoundedByteBufferPool will spew report to the log. Disable the TRACE level and
then check the log. You'll see allocation rate, size of pool, size of buffers in pool, etc.

> [PERF] Reuse the IPCUtil#buildCellBlock buffer
> ----------------------------------------------
>
>                 Key: HBASE-13142
>                 URL: https://issues.apache.org/jira/browse/HBASE-13142
>             Project: HBase
>          Issue Type: Improvement
>          Components: Performance
>            Reporter: stack
>            Assignee: stack
>              Labels: beginner
>             Fix For: 2.0.0, 1.1.0
>
>         Attachments: 13142.txt, 13142v2.txt, 13142v3.txt, 13142v5.txt, traces.2.svg,
traces.svg
>
>
> Running some scan profiling, flight recorder was mildly fingering resize of the buffer
allocated in IPCUtil#buildCellBlock as a point of contention.  It was half-hearted blaming
it for a few hundreds of ms over a five minute sampling with a few tens of instances showing.
> I tried then w/ flamegraph/lightweight profiler and this reported the buffer allocations
taking 22% of our total CPU. See attachment trace.svg.
> I enabled TRACE-level logging on org.apache.hadoop.hbase.ipc.IPCUtil and indeed every
allocation was doing a resize from initial allocation of 16k -- the default up to 220k (this
test returns ten randomly sized rows zipfian sized between 0 and 8k).
> Upping the allocation to 220k meant we now avoided the resize but the initial allocation
was now blamed for 10% of allocations (see trace.2.svg attached).
> Lets do buffer reuse.  Will save a bunch of allocation and CPU.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message