hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Chris Nauroth (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (HDFS-4725) fix HDFS file handle leaks
Date Mon, 22 Apr 2013 23:03:17 GMT

     [ https://issues.apache.org/jira/browse/HDFS-4725?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel

Chris Nauroth updated HDFS-4725:

    Attachment: HDFS-4725.1.patch

I'm attaching a patch to address the file handle leaks that I've found.  Here is a summary
of the changes:

# Namenode shutdown during an early termination (fatal error) did not guarantee complete cleanup.
 I've made changes to {{NameNode}} and {{FSEditLog}} to guarantee all edits logs get closed
and storage lock files get released via {{FSNamesystem#close}}.  Previously, we could skip
closing edits logs and releasing lock files when {{ExitUtil#terminate}} was running in test
mode, and therefore throwing an unchecked exception instead of exiting the process.
# {{OfflineEditsBinaryLoader}} was not closing the edits log it was reading.  I've added a
finally block to guarantee that it gets closed.
# {{DFSTestUtil#loadFile}} had a file handle leak, so I added a finally block to guarantee
# Various tests were not calling shutdown on their {{MiniDFSCluster}} instances.  You'll see
some big diffs from where I added finally blocks to guarantee shutdown, but this is just due
to the indentation changes.
# {{TestEditLogJournalFailures}} had set up a mock to do nothing when anything called {{EditLogOutputStream#abort}}.
 Under normal operation, the file would get closed in that method, so the mocking would cause
tests to leak file handles.  From what I can tell, there is no particular reason that the
tests need to skip the abort, so I just removed that part of the mocking.

I tested this successfully on Mac and Windows.  With this patch, we'll stop seeing failures
on Windows in these test suites:

* {{TestEditLogJournalFailures}}
* {{TestStartup}}
* {{TestDatanodeDeath}}
* {{TestDFSClientExcludedNodes}}
* {{TestDistributedFileSystem}}
* {{TestQuota}}
* {{TestOfflineEditsViewer}}

> fix HDFS file handle leaks
> --------------------------
>                 Key: HDFS-4725
>                 URL: https://issues.apache.org/jira/browse/HDFS-4725
>             Project: Hadoop HDFS
>          Issue Type: Bug
>          Components: namenode, test, tools
>    Affects Versions: 3.0.0
>            Reporter: Chris Nauroth
>            Assignee: Chris Nauroth
>         Attachments: HDFS-4725.1.patch
> The scope of this issue is to fix multiple file handle leaks observed from recent HDFS
test runs.

This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

View raw message