hadoop-common-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Suresh Srinivas (Commented) (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HADOOP-7973) DistributedFileSystem close has severe consequences
Date Fri, 13 Jan 2012 18:24:39 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-7973?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13185733#comment-13185733

Suresh Srinivas commented on HADOOP-7973:

Some comments:
bq. a decision was made on an earlier jira to not add reference counting to cached filesystem
Reference counting will not work. One could get a file system instance and pass it to others
and then close it. This kind of code might exist in the framework.

bq. DistributedFileSystem#close becomes a no-op. The finalizer will close the DFSClient.
What is the finalizer - java finalizer? or you mean garbage collection?

File system cache has been a problem that has been lurking for a while. App is expected to
write the correct code - that is if the file system is closed, do not use it post that. However
I am not sure if apps do handle this correctly.

Other alternative solution is to disable cache. This is already configurable. See the following
snippet in FileSystem.java
    String disableCacheName = String.format("fs.%s.impl.disable.cache", scheme);
    if (conf.getBoolean(disableCacheName, false)) {
      return createFileSystem(uri, conf);

> DistributedFileSystem close has severe consequences
> ---------------------------------------------------
>                 Key: HADOOP-7973
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7973
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: fs
>    Affects Versions: 1.0.0
>            Reporter: Daryn Sharp
>            Assignee: Daryn Sharp
>            Priority: Blocker
>         Attachments: HADOOP-7973.patch
> The way {{FileSystem#close}} works is very problematic.  Since the {{FileSystems}} are
cached, any {{close}} by any caller will cause problems for every other reference to it. 
Will add more detail in the comments.

This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira


View raw message