hbase-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "deepankar (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HBASE-13819) Make RPC layer CellBlock buffer a DirectByteBuffer
Date Wed, 23 Mar 2016 03:18:25 GMT

    [ https://issues.apache.org/jira/browse/HBASE-13819?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15207822#comment-15207822

deepankar commented on HBASE-13819:


We recently pulled this patch internally and are seeing some significant side effects of BoundedByteBufferPool
where the system runs out of direct memory whenever there is some network congestion or some
client side issues. 

System props used :
RPC Handlers = 300
Number of Clients ~200, Threads in each client ~20, we use asynchbase so all requests are
multiplexed on single connections
Initial buffer size is 16 kb
Buffer size settles to 512 kb over time(from debug statements we put in) 
Max number to cache (reservoir.initial.max ) is around 2048 (also tried with 4096)
DirectMem accounted for this (is 2048 * 512 KB + 1 GB)

We took a heapdump and analysed the contents of the heap using VisualVM OQL and we found that

number of rpcs that were queued in the responder was around ~4000 and this leads to exhaustion
of the direct buffer space, digging a little bit more deeper the responses buffers in the
pendingCallsQueue in connection accounted for 3181117440 bytes, even though the real 
response size (buffers are allocated for 512kb even when the response is small) accounted
only for 84451734.0 bytes. 

[~anoop.hbase] suggested that since any way we are using buffer chain to create a CellBlock
it would better to create a new ByteBufferOutputStream which acquires buffers from the pool
instead of allocating a new one with very high moving average and removing the moving average
overall and having a fixed size buffers instead ?

Here are the visualVM query used 
select sum(map(heap.objects('java.nio.DirectByteBuffer'), function (a) {
   var x = 0;
   var callx = null;
   forEachReferrer(function (y) { 
    if (classof(y).name == 'org.apache.hadoop.hbase.ipc.RpcServer$Call') {
       x = -1;
       forEachReferrer(function (px) {
         if (classof(px).name == 'java.util.concurrent.ConcurrentLinkedDeque$Node') {
            callx = y;             
            x = 1;
       }, y);
   }, a);

   if (a.att == null &&  x == 1 && callx.response.bufferOffset == 0 &&
callx.response.remaining != 0) {
      // return callx.response.remaining
      return a.capacity
   } else {
      return  0

> Make RPC layer CellBlock buffer a DirectByteBuffer
> --------------------------------------------------
>                 Key: HBASE-13819
>                 URL: https://issues.apache.org/jira/browse/HBASE-13819
>             Project: HBase
>          Issue Type: Sub-task
>          Components: Scanners
>            Reporter: Anoop Sam John
>            Assignee: Anoop Sam John
>             Fix For: 2.0.0, 1.3.0
>         Attachments: HBASE-13819.patch, HBASE-13819_branch-1.patch, HBASE-13819_branch-1.patch,
> In RPC layer, when we make a cellBlock to put as RPC payload, we will make an on heap
byte buffer (via BoundedByteBufferPool). The pool will keep upto certain number of buffers.
This jira aims at testing possibility for making this buffers off heap ones. (DBB)  The advantages
> 1. Unsafe based writes to off heap is faster than that to on heap. Now we are not using
unsafe based writes at all. Even if we add, DBB will be better
> 2. When Cells are backed by off heap (HBASE-11425) off heap to off heap writes will be
> 3. When checked the code in SocketChannel impl, if we pass a HeapByteBuffer to the socket
channel, it will create a temp DBB and copy data to there and only DBBs will be moved to Sockets.
If we make DBB 1st hand itself, we can  avoid this one more level of copying.
> Will do different perf testing with changed and report back.

This message was sent by Atlassian JIRA

View raw message