hbase-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Todd Lipcon (Commented) (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HBASE-5141) Memory leak in MonitoredRPCHandlerImpl
Date Fri, 06 Jan 2012 23:45:44 GMT

    [ https://issues.apache.org/jira/browse/HBASE-5141?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13181697#comment-13181697
] 

Todd Lipcon commented on HBASE-5141:
------------------------------------

Part of me thinks this is a configuration error -- a conservative admin should always assume
that the memory usage of IPC buffers = numHandlers*maxPayload, since you could always have
the threads all concurrently handling large calls. So memory should be allocated for this.
The fact that the memory is kept around until the next call makes it more likely you'd hit
this, but it's still a potential problem regardless.

I think the proper fix for this problem is to keep an atomic counter for the amount of memory
used by IPC handlers, and gate the read calls off the wire based on a memory budget.
                
> Memory leak in MonitoredRPCHandlerImpl
> --------------------------------------
>
>                 Key: HBASE-5141
>                 URL: https://issues.apache.org/jira/browse/HBASE-5141
>             Project: HBase
>          Issue Type: Bug
>    Affects Versions: 0.92.0
>            Reporter: Jean-Daniel Cryans
>            Priority: Blocker
>             Fix For: 0.92.0, 0.94.0
>
>         Attachments: Screen Shot 2012-01-06 at 3.03.09 PM.png
>
>
> I got a pretty reliable way of OOME'ing my region servers. Using a big payload (64MB
in my case), a default heap and default number of handlers, it's not too long that all the
MonitoredRPCHandlerImpl hold on a 64MB reference and once a compaction kicks in it kills everything.
> The issue is that even after the RPC call is done, the packet still lives in MonitoredRPCHandlerImpl.
> Will attach a screen shot of jprofiler's analysis in a moment.
> This is a blocker for 0.92.0, anyone using a high number of handlers and bigish values
will kill themselves.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

Mime
View raw message