hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Daryn Sharp (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HDFS-3359) DFSClient.close should close cached sockets
Date Mon, 07 May 2012 13:36:53 GMT

    [ https://issues.apache.org/jira/browse/HDFS-3359?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13269609#comment-13269609

Daryn Sharp commented on HDFS-3359:

# streams should be closed by users individually.

Agreed, they _should_ be, but accidents happen which results in leaks.  Closing a {{FileSystem}}
will force a hard close of all streams.  How will {{FileContext}} provide the same functionality?

# client (DFSClient) is just a Java object.
# lease renewer is a global cache so that multiple being-written-files/DFSClients could share
a single thread.

If the client isn't explicitly closed, it appears the lease renewer holds a reference to it
so the client will never be closed until exit.  Will this cause the client's leases to be
renewed until exit?  The renewer will shutdown if all clients are closed, but due to the aforementioned,
this will never happen.  Will this cause the process or a new thread to encounter lease problems
when re-opening the file?

# do you mean deleteOnExit by temp files? Anyway, deleteOnExit is handled by shutdown hooks.

Temp files are removed either by explicit close, or via an implicit close issued by the shutdown
hooks.  How can a long running client force the removal the temp files prior to exit?
> DFSClient.close should close cached sockets
> -------------------------------------------
>                 Key: HDFS-3359
>                 URL: https://issues.apache.org/jira/browse/HDFS-3359
>             Project: Hadoop HDFS
>          Issue Type: Bug
>          Components: hdfs client
>    Affects Versions: 0.22.0, 2.0.0
>            Reporter: Todd Lipcon
>            Assignee: Todd Lipcon
>            Priority: Critical
>             Fix For: 0.23.3, 2.0.0
>         Attachments: hdfs-3359-branch-0.23.txt, hdfs-3359.txt, hdfs-3359.txt
> Some applications like the TT/JT (pre-2.0) and probably the RM/NM cycle through DistributedFileSystem
objects reasonably frequently. So long as they call close() it isn't a big problem, except
that currently DFSClient.close() doesn't explicitly close the SocketCache. So unless a full
GC runs (causing the references to get finalized), many SocketCaches can get orphaned, each
with many open sockets inside. We should fix the close() function to close all cached sockets.

This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira


View raw message