hbase-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Hiroshi Ikeda (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HBASE-14490) [RpcServer] reuse request read buffer
Date Fri, 23 Oct 2015 02:59:27 GMT

    [ https://issues.apache.org/jira/browse/HBASE-14490?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14970333#comment-14970333

Hiroshi Ikeda commented on HBASE-14490:

I know you doubling idea.If we use Pool, we hope fewer number of buffer service more Connection.If
Pool can't reduce amount of buffer dynamically, it will increase to N ("N" is number of Connection
).In that case, there is no different between Pool and non-pool.I'm thinking about Pool that
can control the number of buffer and each size of buffer.

Sorry, could you be more specific? The pool can hold much more number of buffers than connections
on a heavy load, and lower on a low load. It is easy and it would be not bad to leave that
adjustment to weak/soft references (I still recommend to use weak one). Leaving it aside,
from the point of a view of GC (and creating cost), what we should pay attention to is, not
how many buffers we are currently keeping, but how many buffers we have discarded (and created).
Non-pooling creates and discards a buffer each time and pooling would have an advantage.

It maintain average size.It expect hitting 50% req.

If it is much more possibility that coming data has a different size, it nearly discards the
buffer each time, which depends on the protocol HBase uses. For example, receiving data whose
size of 100, 1000, 100, 1000, ..., the average is 500-600 and your logic creates a buffer
of the average size or 1000 for each data.

> [RpcServer] reuse request read buffer
> -------------------------------------
>                 Key: HBASE-14490
>                 URL: https://issues.apache.org/jira/browse/HBASE-14490
>             Project: HBase
>          Issue Type: Improvement
>          Components: IPC/RPC
>    Affects Versions: 2.0.0, 1.0.2
>            Reporter: Zephyr Guo
>            Assignee: Zephyr Guo
>              Labels: performance
>             Fix For: 2.0.0, 1.0.2
>         Attachments: ByteBufferPool.java, HBASE-14490-v1.patch, HBASE-14490-v10.patch,
HBASE-14490-v11.patch, HBASE-14490-v2.patch, HBASE-14490-v3.patch, HBASE-14490-v4.patch, HBASE-14490-v5.patch,
HBASE-14490-v6.patch, HBASE-14490-v7.patch, HBASE-14490-v8.patch, HBASE-14490-v9.patch
> Reuse buffer to read request.It's not necessary to every request free buffer.The idea
of optimization is to reduce the times that allocate ByteBuffer.
> *Modification*
> 1. {{saslReadAndProcess}} ,{{processOneRpc}}, {{processUnwrappedData}}, {{processConnectionHeader}}
accept a ByteBuffer instead of byte[].They can move {{ByteBuffer.position}} correctly when
we have read the data.
> 2. {{processUnwrappedData}} no longer use any extra memory.
> 3. Maintaining a reused ByteBuffer in each {{Connection}}.
> a) Size is dynamic to fit specific client.
> b) If size deviate far from average size, we replace a new one.
> c) Using a SoftReference to reference the buffer.

This message was sent by Atlassian JIRA

View raw message