hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Chris Nauroth (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (HDFS-4725) fix HDFS file handle leaks
Date Mon, 22 Apr 2013 23:03:17 GMT

     [ https://issues.apache.org/jira/browse/HDFS-4725?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Chris Nauroth updated HDFS-4725:
--------------------------------

    Attachment: HDFS-4725.1.patch

I'm attaching a patch to address the file handle leaks that I've found.  Here is a summary
of the changes:

# Namenode shutdown during an early termination (fatal error) did not guarantee complete cleanup.
 I've made changes to {{NameNode}} and {{FSEditLog}} to guarantee all edits logs get closed
and storage lock files get released via {{FSNamesystem#close}}.  Previously, we could skip
closing edits logs and releasing lock files when {{ExitUtil#terminate}} was running in test
mode, and therefore throwing an unchecked exception instead of exiting the process.
# {{OfflineEditsBinaryLoader}} was not closing the edits log it was reading.  I've added a
finally block to guarantee that it gets closed.
# {{DFSTestUtil#loadFile}} had a file handle leak, so I added a finally block to guarantee
close.
# Various tests were not calling shutdown on their {{MiniDFSCluster}} instances.  You'll see
some big diffs from where I added finally blocks to guarantee shutdown, but this is just due
to the indentation changes.
# {{TestEditLogJournalFailures}} had set up a mock to do nothing when anything called {{EditLogOutputStream#abort}}.
 Under normal operation, the file would get closed in that method, so the mocking would cause
tests to leak file handles.  From what I can tell, there is no particular reason that the
tests need to skip the abort, so I just removed that part of the mocking.

I tested this successfully on Mac and Windows.  With this patch, we'll stop seeing failures
on Windows in these test suites:

* {{TestEditLogJournalFailures}}
* {{TestStartup}}
* {{TestDatanodeDeath}}
* {{TestDFSClientExcludedNodes}}
* {{TestDistributedFileSystem}}
* {{TestQuota}}
* {{TestOfflineEditsViewer}}

                
> fix HDFS file handle leaks
> --------------------------
>
>                 Key: HDFS-4725
>                 URL: https://issues.apache.org/jira/browse/HDFS-4725
>             Project: Hadoop HDFS
>          Issue Type: Bug
>          Components: namenode, test, tools
>    Affects Versions: 3.0.0
>            Reporter: Chris Nauroth
>            Assignee: Chris Nauroth
>         Attachments: HDFS-4725.1.patch
>
>
> The scope of this issue is to fix multiple file handle leaks observed from recent HDFS
test runs.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

Mime
View raw message