hbase-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "deepankar (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HBASE-13819) Make RPC layer CellBlock buffer a DirectByteBuffer
Date Wed, 23 Mar 2016 23:26:25 GMT

    [ https://issues.apache.org/jira/browse/HBASE-13819?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15209409#comment-15209409
] 

deepankar commented on HBASE-13819:
-----------------------------------

bq. Which Q we talking? The is config to put bounds on the call queue where we will reject
calls...

The one that reject calls only accounts for incoming request sizes and there is default 2GB
limit I think. The response sizes are not accounted in this I think.
The Queue is RpcServer$Connection.responseQueue.

bq. Yeah, there is heuristic and we grow till we hit an average. Are we saying we grew to
512k and then afterward, all calls were 16k only? Is this a problem?

In our use case our worst case response size is 512 kb (hard capped from client side) and
our avg response size is between 12kb, what we observe is after 3 - 4 days of running almost
always the moving avg is at 512 kb and in the heap dump all the response buffers is of size
512 kb

> Make RPC layer CellBlock buffer a DirectByteBuffer
> --------------------------------------------------
>
>                 Key: HBASE-13819
>                 URL: https://issues.apache.org/jira/browse/HBASE-13819
>             Project: HBase
>          Issue Type: Sub-task
>          Components: Scanners
>            Reporter: Anoop Sam John
>            Assignee: Anoop Sam John
>             Fix For: 2.0.0, 1.3.0
>
>         Attachments: HBASE-13819.patch, HBASE-13819_branch-1.patch, HBASE-13819_branch-1.patch,
HBASE-13819_branch-1.patch
>
>
> In RPC layer, when we make a cellBlock to put as RPC payload, we will make an on heap
byte buffer (via BoundedByteBufferPool). The pool will keep upto certain number of buffers.
This jira aims at testing possibility for making this buffers off heap ones. (DBB)  The advantages
> 1. Unsafe based writes to off heap is faster than that to on heap. Now we are not using
unsafe based writes at all. Even if we add, DBB will be better
> 2. When Cells are backed by off heap (HBASE-11425) off heap to off heap writes will be
better
> 3. When checked the code in SocketChannel impl, if we pass a HeapByteBuffer to the socket
channel, it will create a temp DBB and copy data to there and only DBBs will be moved to Sockets.
If we make DBB 1st hand itself, we can  avoid this one more level of copying.
> Will do different perf testing with changed and report back.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message