hbase-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Asaf Mesika (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HBASE-4633) Potential memory leak in client RPC timeout mechanism
Date Wed, 07 Aug 2013 04:45:52 GMT

    [ https://issues.apache.org/jira/browse/HBASE-4633?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13731633#comment-13731633
] 

Asaf Mesika commented on HBASE-4633:
------------------------------------

If running with -Xmx6GB our application running HBase Client, then we are witnessing process
resident memory increasing to 10GB. When placing -X:MaxDirectMemorySize=2g, we're seeing WARN
OutOfMemoryException, but no error since the client has 10 retries. Our next step is to run
it with profiler to diagnose why so much direct memory is used. Want to make sure its HBaseClient
code and not some other third party lib we're using (although we're pretty sure its the only
one calling ByteBuffer.allocateDirect transitively).

                
> Potential memory leak in client RPC timeout mechanism
> -----------------------------------------------------
>
>                 Key: HBASE-4633
>                 URL: https://issues.apache.org/jira/browse/HBASE-4633
>             Project: HBase
>          Issue Type: Bug
>          Components: Client
>    Affects Versions: 0.90.3
>         Environment: HBase version: 0.90.3 + Patches , Hadoop version: CDH3u0
>            Reporter: Shrijeet Paliwal
>         Attachments: HBaseclientstack.png
>
>
> Relevant Jiras: https://issues.apache.org/jira/browse/HBASE-2937,
> https://issues.apache.org/jira/browse/HBASE-4003
> We have been using the 'hbase.client.operation.timeout' knob
> introduced in 2937 for quite some time now. It helps us enforce SLA.
> We have two HBase clusters and two HBase client clusters. One of them
> is much busier than the other.
> We have seen a deterministic behavior of clients running in busy
> cluster. Their (client's) memory footprint increases consistently
> after they have been up for roughly 24 hours.
> This memory footprint almost doubles from its usual value (usual case
> == RPC timeout disabled). After much investigation nothing concrete
> came out and we had to put a hack
> which keep heap size in control even when RPC timeout is enabled. Also
> note , the same behavior is not observed in 'not so busy
> cluster.
> The patch is here : https://gist.github.com/1288023

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

Mime
View raw message