hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Daryn Sharp (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HDFS-3646) LeaseRenewer can hold reference to inactive DFSClient instances forever
Date Thu, 12 Jul 2012 16:35:34 GMT

    [ https://issues.apache.org/jira/browse/HDFS-3646?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13412916#comment-13412916

Daryn Sharp commented on HDFS-3646:

bq. IMO, this will a leak from application side, since there is a bug in closing the streams
from app.

Agreed, but it can have pretty severe consequences that aren't easily detected unless explicitly
hunting for leaks.  It makes me uneasy that an out of scope fs stream can cause a massive
leak of heavy objects, threads, and tie up sockets that may exhaust fds and/or memory for
long running processes.  Emitting an angry log error for lost & unclosed streams may be
more beneficial. 

I don't think a finalizer on the fs will work.  If I do {{in = path.getFileSystem(conf).open(...)}},
the fs might get garbage collected but we certainly don't want its finalizer to shoot the
dfs client that is still holding open a stream.  Maybe a finalizer on the dfs client, but
in any case, the circular hard references need to be broken somehow.
> LeaseRenewer can hold reference to inactive DFSClient instances forever
> -----------------------------------------------------------------------
>                 Key: HDFS-3646
>                 URL: https://issues.apache.org/jira/browse/HDFS-3646
>             Project: Hadoop HDFS
>          Issue Type: Bug
>          Components: hdfs client
>    Affects Versions: 0.23.3, 2.0.0-alpha
>            Reporter: Kihwal Lee
>            Assignee: Kihwal Lee
>            Priority: Critical
>             Fix For: 0.23.3, 2.0.1-alpha, 3.0.0
> If {{LeaseRenewer#closeClient()}} is not called, {{LeaseRenewer}} keeps the reference
to a {{DFSClient}} instance in {{dfsclients}} forever. This prevents {{DFSClient}}, {{LeaseRenewer}},
conf, etc. from being garbage collected, leading to memory leak.
> {{LeaseRenewer}} should remove the reference after some delay, if a {{DFSClient}} instance
no longer has active streams.

This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira


View raw message