hadoop-common-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Owen O'Malley (JIRA)" <j...@apache.org>
Subject [jira] Commented: (HADOOP-1130) Remove unused ClientFinalizer in DFSClient
Date Tue, 27 Mar 2007 16:55:32 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-1130?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12484500
] 

Owen O'Malley commented on HADOOP-1130:
---------------------------------------

Jira keeps eating my comments on this issue. *sigh*

> As hadoop may be used as a third party library, it shouldn't install shutdown hook by
itself.

Actually, I think you have this backwards. Since Hadoop may be used as a third party library,
it must use finalizers and shutdown hooks to clean up properly. The problems with have an
explicit close method that the user must call are:
  1. Forcing users to remember to cleanup your library is error-prone.
  2. Just because library Foo is done does not mean library Bar is done and Bar will lose
when Foo calls close.

That said, I think that with some relatively minor changes, we can get the effect that you
want (minimal garbage after the usage of the FileSystem is done):
  1. Fix HADOOP-1160 so that the FileSystem.closeAll actually closes the client.
  2. Have FileSystem.closeAll clear the CACHE map via reset.
  3. Have the DFSClient.close method remove the client from the ClientFinalizer.
  4. Have the constructor add "this" to the ClientFinalizer.

So after the closeAll call, you'll still have the DFSClient class loaded, but not much else.
In particular, you'll have no instances of DFSClient left.

> Remove unused ClientFinalizer in DFSClient
> ------------------------------------------
>
>                 Key: HADOOP-1130
>                 URL: https://issues.apache.org/jira/browse/HADOOP-1130
>             Project: Hadoop
>          Issue Type: Bug
>          Components: dfs
>    Affects Versions: 0.10.1, 0.11.0, 0.11.1, 0.11.2, 0.12.0
>            Reporter: Philippe Gassmann
>            Priority: Critical
>         Attachments: HADOOP-1130.diff
>
>
> The ClientFinalizer shutdown hook is not used.
> This can lead to severe memory leaks if you use the DFSClient in a dynamic class loading
context (such as in a webapp) as the DFSClient.ClientFinalizer class is retained in the memory
by the shutdown hook. It retains also the current ClassLoader and thus all classes loaded
by the ClassLoader. (Threads put in the shutdown hook are never garbage collected).

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


Mime
View raw message