hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Kihwal Lee (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HDFS-3646) LeaseRenewer can hold reference to inactive DFSClient instances forever
Date Tue, 17 Jul 2012 16:27:34 GMT

    [ https://issues.apache.org/jira/browse/HDFS-3646?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13416339#comment-13416339

Kihwal Lee commented on HDFS-3646:

Since we are now removing the reference to a client when all streams are closed, the renwer's
loop exits right away ({{clientsRunning()}} is false when there is no more clients left).
In order to honor the grace period, I had to move the check and set {{emptyTime}} so that
the thread is kept alive until isRenewerExpired() returns true.

The failed test cases don't seem to be related to the patch and I could not reproduce the

- TestFileAppend4 ran sucessfully, but it timed out because Datanode didn't shutdown.
012-07-17 01:58:56,034 INFO  datanode.DataNode (DataNode.java:shutdown(1079)) 
- Waiting for threadgroup to exit, active threads is 1
- TestBlocksWithNotEnoughRacks: MiniDFSCluster exited in the middle of test.

Regarding null authority, there is a check for nameNodeUri being null, it will throw an IllegalArgumentException.
If the uri is not null, but the authority is null, it also won't go far because proxy won't
get created properly. Since {{DFSClient}} blows up anyway well before a {LeaseRenewer} is
created, the content of variable, {authority}, being "null" or null does not matter.

I will upload the updated patch soon.
> LeaseRenewer can hold reference to inactive DFSClient instances forever
> -----------------------------------------------------------------------
>                 Key: HDFS-3646
>                 URL: https://issues.apache.org/jira/browse/HDFS-3646
>             Project: Hadoop HDFS
>          Issue Type: Bug
>          Components: hdfs client
>    Affects Versions: 0.23.3, 2.0.0-alpha
>            Reporter: Kihwal Lee
>            Assignee: Kihwal Lee
>            Priority: Critical
>             Fix For: 0.23.3, 2.1.0-alpha, 3.0.0
>         Attachments: hdfs-3646.patch, hdfs-3646.patch.txt
> If {{LeaseRenewer#closeClient()}} is not called, {{LeaseRenewer}} keeps the reference
to a {{DFSClient}} instance in {{dfsclients}} forever. This prevents {{DFSClient}}, {{LeaseRenewer}},
conf, etc. from being garbage collected, leading to memory leak.
> {{LeaseRenewer}} should remove the reference after some delay, if a {{DFSClient}} instance
no longer has active streams.

This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira


View raw message