hbase-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Enis Soztutar (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HBASE-15177) Reduce garbage created under high load
Date Wed, 27 Jan 2016 19:30:39 GMT

    [ https://issues.apache.org/jira/browse/HBASE-15177?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15120021#comment-15120021
] 

Enis Soztutar commented on HBASE-15177:
---------------------------------------

Thanks for checking. 
bq. So the reason for going over to BB and BBInputStream is for supporting offheap BB and
BB reuse right?
Yes that was the idea, but the patch does not contain any changes for this. I have tried with
offheap IPC BBs with and without reuse, but the problem is with PB's CodedInputStream.

This allocates 4K buffer and copy bytes into this buffer when using InputStream over DBB.

{code}
  private CodedInputStream(final InputStream input) {
    buffer = new byte[BUFFER_SIZE];
    bufferSize = 0;
    bufferPos = 0;
    totalBytesRetired = 0;
    this.input = input;
  }
{code} 

Because of this extra 4K buffer allocation and extra copying of the bytes from offheap to
on-heap, the benefits of reading into off-heap buffer is not realized and there is way more
garbage created because of this. Also even if you re-use offheap BBs, CodedInputStream will
allocate 4K byte[]'s anyway. I am testing with gets where the rpc request size is obviously
much smaller than 4K. Maybe we need to re-write CodedInputStream? 

bq. But does that directly have an impact on the GC for now?
We are creating 2 CodedInputStreams today for reading each RPC call. This patch saves on 1.


bq. This change from 'null' to 'controller' may not be really needed for the close() scanner
call?
We are doing a close scanner RPC, and it should also have a priority associated. 

bq. We are doing some PoC in this area. Could not test it for perf and that is why did not
raise any Jira. The idea is extension what HBASE-14490. says. We read reqs into off heap ByteBuffers
that come from pool.
Would be good to have this, but we have to be careful about the CIS as above. 

bq. In the patch there is some changes around setting priority as well. Intended? I did not
see the patch in detail.
Yes, this is to save on the unnecessary garbage being created from AnnotationReadingPriorityFunction.
We already set the priority from the client side for gets and multis. But it was only used
on the server side if it is a multi. This patch fixes that, and also makes sure that scanner
open and close sets the priority as well. 






> Reduce garbage created under high load
> --------------------------------------
>
>                 Key: HBASE-15177
>                 URL: https://issues.apache.org/jira/browse/HBASE-15177
>             Project: HBase
>          Issue Type: Improvement
>            Reporter: Enis Soztutar
>            Assignee: Enis Soztutar
>             Fix For: 2.0.0, 1.3.0
>
>         Attachments: Screen Shot 2016-01-26 at 10.03.48 PM.png, Screen Shot 2016-01-26
at 10.03.56 PM.png, Screen Shot 2016-01-26 at 10.06.16 PM.png, Screen Shot 2016-01-26 at 10.15.15
PM.png, hbase-15177_v0.patch
>
>
> I have been doing some profiling of the garbage being created. The idea was to follow
up on HBASE-14490 and experiment with offheap IPC byte buffers and byte buffer re-use. However,
without changing the IPC byte buffers for now, there are a couple of (easy) improvements that
I've identified from profiling: 
> 1. RPCServer.Connection.processRequest() should work with ByteBuffer instead of byte[]
and not-recreate CodedInputStream a few times. 
> 2. RSRpcServices.getRegion() allocates two byte arrays for region, while only 1 is needed.
> 3. AnnotationReadingPriorityFunction is very expensive in allocations. Mainly it allocates
the regionName byte[] to get the table name. We already set the priority for most of the operations
(multi, get, increment, etc) but we are only reading the priority in case of multi. We should
use the priority from the client side. 
> Lets do the simple improvements in this patch, we can get to IPC buffer re-use in HBASE-14490.




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message