hbase-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Duo Zhang (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HBASE-16635) RpcClient under heavy load leaks some netty bytebuf
Date Thu, 15 Sep 2016 07:18:20 GMT

    [ https://issues.apache.org/jira/browse/HBASE-16635?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15492619#comment-15492619
] 

Duo Zhang commented on HBASE-16635:
-----------------------------------

Ah after a careful review, I found some leaks...

First is in SaslWrapHandler, I forgot to release the ByteBuf after read all its content to
a byte array.

Second, the two connection header ByteBufs in NettyRpcConnection. It is allocated by Unpooled
so I ignored the release intentionally when writing down the code, maybe. But after reading
the code of netty, I believe the ByteBufs allocated by Unpooled should also be released...

I do not know why I have an illusion that an unpooled ByteBuf does not need to be released.
I had hit a 'buffer already released' problem when I forgot to use retainedDuplicate before
writing out the header. This definitely means netty will release unpooled ByteBuf...

Please tell me if you have other findings. Thanks. My fault.

> RpcClient under heavy load leaks some netty bytebuf
> ---------------------------------------------------
>
>                 Key: HBASE-16635
>                 URL: https://issues.apache.org/jira/browse/HBASE-16635
>             Project: HBase
>          Issue Type: Bug
>            Reporter: ramkrishna.s.vasudevan
>            Assignee: ramkrishna.s.vasudevan
>            Priority: Minor
>             Fix For: 2.0.0
>
>
> Yet to analyse the actual root cause. 
> But the case is that when we run a PE tool with 50 threads under heavy load when the
writes are clogged I think we have some netty Bytebuf leak. Not sure if it is a serious issue
but we get this log
> {code}
> 2016-09-14 19:37:09,767 ERROR [Default-IPC-NioEventLoopGroup-1-16] util.ResourceLeakDetector:
LEAK: ByteBuf.release() was not called before it's garbage-collected. Enable advanced leak
reporting to find out where the leak occurred. To enable advanced leak reporting, specify
the JVM option '-Dio.netty.leakDetection.level=advanced' or call ResourceLeakDetector.setLevel()
See http://netty.io/wiki/reference-counted-objects.html for more information.
> {code}
> So reading the given link it is because of some ByteBuf that was not released properly
by the client and hence it gets GCed automatically. Netty provides tips and tricks to find
the root cause. Will get back here.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message