hbase-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Shrijeet Paliwal (Commented) (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HBASE-4633) Potential memory leak in client RPC timeout mechanism
Date Thu, 20 Oct 2011 17:19:11 GMT

    [ https://issues.apache.org/jira/browse/HBASE-4633?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13131786#comment-13131786
] 

Shrijeet Paliwal commented on HBASE-4633:
-----------------------------------------

@Stack
No we did not run with that flag. Also we never got to a point when application had to die
cause of OOM. The reasons (I guess) are :
# We have GC flags to do garbage collection as fast as possible. 
# The monitoring in place starts sending our alerts and we usually shoot the server in the
head before it OOMs
# The load balancer will kick in and start sending no work to application server realizing
it is in bad state. 

As mentioned earlier I have found it hard to reproduce in dev environment, failing to simulate
production like load. But I must try again when.
                
> Potential memory leak in client RPC timeout mechanism
> -----------------------------------------------------
>
>                 Key: HBASE-4633
>                 URL: https://issues.apache.org/jira/browse/HBASE-4633
>             Project: HBase
>          Issue Type: Bug
>          Components: client
>    Affects Versions: 0.90.3
>         Environment: HBase version: 0.90.3 + Patches , Hadoop version: CDH3u0
>            Reporter: Shrijeet Paliwal
>
> Relevant Jiras: https://issues.apache.org/jira/browse/HBASE-2937,
> https://issues.apache.org/jira/browse/HBASE-4003
> We have been using the 'hbase.client.operation.timeout' knob
> introduced in 2937 for quite some time now. It helps us enforce SLA.
> We have two HBase clusters and two HBase client clusters. One of them
> is much busier than the other.
> We have seen a deterministic behavior of clients running in busy
> cluster. Their (client's) memory footprint increases consistently
> after they have been up for roughly 24 hours.
> This memory footprint almost doubles from its usual value (usual case
> == RPC timeout disabled). After much investigation nothing concrete
> came out and we had to put a hack
> which keep heap size in control even when RPC timeout is enabled. Also
> note , the same behavior is not observed in 'not so busy
> cluster.
> The patch is here : https://gist.github.com/1288023

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

Mime
View raw message