hbase-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "stack (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HBASE-13819) Make RPC layer CellBlock buffer a DirectByteBuffer
Date Wed, 23 Mar 2016 23:29:25 GMT

    [ https://issues.apache.org/jira/browse/HBASE-13819?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15209412#comment-15209412

stack commented on HBASE-13819:

bq. What Anoop Sam John said is correct, in our case avg response sizes are around 10 -16
kb but the max size is 512kb if you run for significantly longer time the running avg goest
toward 512kb. 
I'd think this a good thing, no? You are seeing some traffic that is 512kb... and the BBBP
can accommodate without a resize. If you do find it a problem, change "hbase.ipc.server.reservoir.max.buffer.size".
 Its default is  1024 * 1024.

bq. Another point its not like that there are lot of request if you have around 200 clients
and lets say .... for these 4000 requests is only 82 MB its atleast 1 - 2 orders of magnitude
of space getting occupied extra.

This sounds like we are leaking? The JVM publishes how much memory is being used in the jmx
metric java.nio.type=BufferPool,name=direct  Does this grow over time?

> Make RPC layer CellBlock buffer a DirectByteBuffer
> --------------------------------------------------
>                 Key: HBASE-13819
>                 URL: https://issues.apache.org/jira/browse/HBASE-13819
>             Project: HBase
>          Issue Type: Sub-task
>          Components: Scanners
>            Reporter: Anoop Sam John
>            Assignee: Anoop Sam John
>             Fix For: 2.0.0, 1.3.0
>         Attachments: HBASE-13819.patch, HBASE-13819_branch-1.patch, HBASE-13819_branch-1.patch,
> In RPC layer, when we make a cellBlock to put as RPC payload, we will make an on heap
byte buffer (via BoundedByteBufferPool). The pool will keep upto certain number of buffers.
This jira aims at testing possibility for making this buffers off heap ones. (DBB)  The advantages
> 1. Unsafe based writes to off heap is faster than that to on heap. Now we are not using
unsafe based writes at all. Even if we add, DBB will be better
> 2. When Cells are backed by off heap (HBASE-11425) off heap to off heap writes will be
> 3. When checked the code in SocketChannel impl, if we pass a HeapByteBuffer to the socket
channel, it will create a temp DBB and copy data to there and only DBBs will be moved to Sockets.
If we make DBB 1st hand itself, we can  avoid this one more level of copying.
> Will do different perf testing with changed and report back.

This message was sent by Atlassian JIRA

View raw message