hbase-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Shrijeet Paliwal (Commented) (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HBASE-4633) Potential memory leak in client RPC timeout mechanism
Date Thu, 20 Oct 2011 15:02:10 GMT

    [ https://issues.apache.org/jira/browse/HBASE-4633?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13131677#comment-13131677
] 

Shrijeet Paliwal commented on HBASE-4633:
-----------------------------------------

@Liyin, 
Are you using RPC timeouts for client operations? 

bq. But Not sure the leak comes from HBase Client jar itself or just our client code. 
In the absence of a concrete evidence that leak is indeed in HBase client jar, I have similar
feeling. It could be in our application layer. 

bq. Our symptom is that the memory footprint will increase as time. But the actual heap size
of the client is not increasing.
We observe the used memory using a collectd plugin http://collectd.org/wiki/index.php/Plugin:Memory

bq. So I am very interested to know when you have keep the heap size in control, is the memory
leaking solved ?
We run with max and min memory set as -Xmx2{X}G -Xms{X}G . And when 'leak' happens the plugin
shows the used memory touching 2X value, so it does seem heap size is increasing. Correct
me here if I am mistaken. 

Let me know if you need more inputs. 
                
> Potential memory leak in client RPC timeout mechanism
> -----------------------------------------------------
>
>                 Key: HBASE-4633
>                 URL: https://issues.apache.org/jira/browse/HBASE-4633
>             Project: HBase
>          Issue Type: Bug
>          Components: client
>    Affects Versions: 0.90.3
>         Environment: HBase version: 0.90.3 + Patches , Hadoop version: CDH3u0
>            Reporter: Shrijeet Paliwal
>
> Relevant Jiras: https://issues.apache.org/jira/browse/HBASE-2937,
> https://issues.apache.org/jira/browse/HBASE-4003
> We have been using the 'hbase.client.operation.timeout' knob
> introduced in 2937 for quite some time now. It helps us enforce SLA.
> We have two HBase clusters and two HBase client clusters. One of them
> is much busier than the other.
> We have seen a deterministic behavior of clients running in busy
> cluster. Their (client's) memory footprint increases consistently
> after they have been up for roughly 24 hours.
> This memory footprint almost doubles from its usual value (usual case
> == RPC timeout disabled). After much investigation nothing concrete
> came out and we had to put a hack
> which keep heap size in control even when RPC timeout is enabled. Also
> note , the same behavior is not observed in 'not so busy
> cluster.
> The patch is here : https://gist.github.com/1288023

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

Mime
View raw message