hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Kihwal Lee (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HDFS-3646) LeaseRenewer can hold reference to inactive DFSClient instances forever
Date Sat, 14 Jul 2012 04:42:35 GMT

    [ https://issues.apache.org/jira/browse/HDFS-3646?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13414295#comment-13414295

Kihwal Lee commented on HDFS-3646:

This patch will allow LeaseRenewer and DFSClient to be GC'ed when all streams are closed by
users. Oozie once reported an OOM due to this problem. I looked at the heap dump and there
were a large number of trios: DFSClient, JobConf and the array entry in LeaseRenewer. There
were no leaked streams/sockets, or FileSystem cache bloat. Had they given up references to
others, they would have been GC'ed. (there was no other external reference to them)

What Daryn mentioned will solve a diffetent problem, which can happen when streams are lost
without being closed. Daryn has some ideas, so he will probably file a separate jira once
its feasibility checks out. 
> LeaseRenewer can hold reference to inactive DFSClient instances forever
> -----------------------------------------------------------------------
>                 Key: HDFS-3646
>                 URL: https://issues.apache.org/jira/browse/HDFS-3646
>             Project: Hadoop HDFS
>          Issue Type: Bug
>          Components: hdfs client
>    Affects Versions: 0.23.3, 2.0.0-alpha
>            Reporter: Kihwal Lee
>            Assignee: Kihwal Lee
>            Priority: Critical
>             Fix For: 0.23.3, 2.0.1-alpha, 3.0.0
>         Attachments: hdfs-3646.patch
> If {{LeaseRenewer#closeClient()}} is not called, {{LeaseRenewer}} keeps the reference
to a {{DFSClient}} instance in {{dfsclients}} forever. This prevents {{DFSClient}}, {{LeaseRenewer}},
conf, etc. from being garbage collected, leading to memory leak.
> {{LeaseRenewer}} should remove the reference after some delay, if a {{DFSClient}} instance
no longer has active streams.

This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira


View raw message