hbase-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Anoop Sam John (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HBASE-13819) Make RPC layer CellBlock buffer a DirectByteBuffer
Date Thu, 24 Mar 2016 04:34:25 GMT

    [ https://issues.apache.org/jira/browse/HBASE-13819?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15209732#comment-15209732
] 

Anoop Sam John commented on HBASE-13819:
----------------------------------------

Its not case of leak Stack..   
Ya I agree that when there is async reqs coming in and the response is some how delayed, there
should be some sort of throttling. Client side can do?
The issue why we go out of off heap memory is because of 2 main reasons
1. The pool has already reached its max capacity of #BBs. And at a given point of time, all
are in use.  Again other Calls ask pool for BBs for their cell block creation.  The pool happily
make new BBs which are off heap with each having size of avg running length.  And all these
cell blocks are tied to Call until the Responder write them to socket.  Ya we wont be keeping
them in pool. But it is kept as is for loner time specially when the response Q is growing.
 
2. Even when the response CellBlock need is very low like 12 KB or so, we waste 512 MB per
response. Waste in the sense that all the 500 MB is not usable at all.  And even the new BBs
which pool create on demand (These might not pooled at all as we reach max #BBs in pool) also
takes 512 MB per BB.

So in a simple way we can say that its really difficult for the user to predict how much max
off heap size he need to give.  With [~dvdreddy] case, he is applying some calc based on the
max #BBs in pool and max BB size + some additional GBs and set the max off heap size as 5
GB.  But this is going wrong..

To explain it with an eg:
Consider one configured the max #BBs in pool as 100.  And max per item size as 1MB. Means
max can have 100 MB off heap consumption by this pool..   Now consider there are lots of reqs
and the response Q is big..  Say the 1st 100 response use all BBs from pool. Now again reqs
are there and say there are like 100 more adding to Q..   Each one req to pool.  It makes
BB off heap for those.   Means out of the pool we have made double the total max size what
we thought it will take..     I agree that we wont store those all BBs in pool and ya the
GC may be able to clean it also..  But for some time (untill we clear these response Q) the
usage is more.

And one more thing for GC is that the full GC only can clean the off heap area?   So this
in other words cause more full GCs? (If we go out
of space in off heap area)!!!

So that is why my thinking abt changing these temp BB creation when happens, those should
be HBBs.

We need to make pool such that we will give a BB back if it is having a free one.  When it
is not having a free one and capacity is not reached, it makes a new DBB and return. If that
is also not the case it wont return any.   The BBBPool will make and take back offf heap BBs
only.  If it can not give, let the caller do what they want (Make on heap BB and make sure
dont give back to pool)

And abt fixing the size of BBs from pool..  Will write in another comment.  This is too big
already :-)

> Make RPC layer CellBlock buffer a DirectByteBuffer
> --------------------------------------------------
>
>                 Key: HBASE-13819
>                 URL: https://issues.apache.org/jira/browse/HBASE-13819
>             Project: HBase
>          Issue Type: Sub-task
>          Components: Scanners
>            Reporter: Anoop Sam John
>            Assignee: Anoop Sam John
>             Fix For: 2.0.0, 1.3.0
>
>         Attachments: HBASE-13819.patch, HBASE-13819_branch-1.patch, HBASE-13819_branch-1.patch,
HBASE-13819_branch-1.patch, q.png
>
>
> In RPC layer, when we make a cellBlock to put as RPC payload, we will make an on heap
byte buffer (via BoundedByteBufferPool). The pool will keep upto certain number of buffers.
This jira aims at testing possibility for making this buffers off heap ones. (DBB)  The advantages
> 1. Unsafe based writes to off heap is faster than that to on heap. Now we are not using
unsafe based writes at all. Even if we add, DBB will be better
> 2. When Cells are backed by off heap (HBASE-11425) off heap to off heap writes will be
better
> 3. When checked the code in SocketChannel impl, if we pass a HeapByteBuffer to the socket
channel, it will create a temp DBB and copy data to there and only DBBs will be moved to Sockets.
If we make DBB 1st hand itself, we can  avoid this one more level of copying.
> Will do different perf testing with changed and report back.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message