hbase-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Anoop Sam John (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HBASE-15525) OutOfMemory could occur when using BoundedByteBufferPool during RPC bursts
Date Thu, 24 Mar 2016 05:20:25 GMT

    [ https://issues.apache.org/jira/browse/HBASE-15525?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15209785#comment-15209785

Anoop Sam John commented on HBASE-15525:

For writing the CellBlock response, we really dont need the whole of it in one BB.   While
writing the response to socket, we have a BufferChain. The Cell block can also be backed by
N BBs rather than 1.   Even within BufferChain, can see that the actual write to socket happens
as chunks of 64 KB size.  So if we have a cell block of 512 KB and instead of one BB, we have
4 BBs of size 128 KB each,  the socket write wont take more time as the total #writes in both
case will be same.

With a model of writing cell block to N smaller sized BBs, we can avoid the need for regrow
(In BBOutputStream) and copy from old smaller sized BB to new BB.  We need a new BBOS which
can do this way of write.   This one will passed to Encoder for writing the Cells to form
cell block.

> OutOfMemory could occur when using BoundedByteBufferPool during RPC bursts
> --------------------------------------------------------------------------
>                 Key: HBASE-15525
>                 URL: https://issues.apache.org/jira/browse/HBASE-15525
>             Project: HBase
>          Issue Type: Bug
>          Components: IPC/RPC
>            Reporter: deepankar
>            Assignee: Anoop Sam John
>            Priority: Critical
> After HBASE-13819 the system some times run out of direct memory whenever there is some
network congestion or some client side issues.
> This was because of pending RPCs in the RPCServer$Connection.responseQueue and since
all the responses in this queue hold a buffer for cellblock from BoundedByteBufferPool this
could takeup a lot of memory if the BoundedByteBufferPool's moving average settles down towards
a higher value 
> See the discussion here [HBASE-13819-comment|https://issues.apache.org/jira/browse/HBASE-13819?focusedCommentId=15207822&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15207822]

This message was sent by Atlassian JIRA

View raw message