hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Mingliang Liu (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (HDFS-10383) Safely close resources in DFSTestUtil
Date Mon, 16 May 2016 18:40:13 GMT

     [ https://issues.apache.org/jira/browse/HDFS-10383?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel

Mingliang Liu updated HDFS-10383:
    Attachment: HDFS-10383.003.patch

Thank you [~walter.k.su] for your helpful suggestion.

Although it's not strictly related to the switch to try-with-resource statement, I think it's
still a case of closing the resources safely (as this jira's title indicates). As a result,
I think it's doable in this jira. I'll update the description of this jira slightly in case
someone, who skips the discussion in comment section, blames us. :)

I like the idea of calling the create RPC directly instead of wrapping a output stream after
creating the file implicitly. In this test helper method, we don't really operate the stream
for write, and thus wrapping a stream is a burden instead of a must-do-work which is to protect
the block data's integrity.

Attached v3 patch.

> Safely close resources in DFSTestUtil
> -------------------------------------
>                 Key: HDFS-10383
>                 URL: https://issues.apache.org/jira/browse/HDFS-10383
>             Project: Hadoop HDFS
>          Issue Type: Improvement
>          Components: test
>            Reporter: Mingliang Liu
>            Assignee: Mingliang Liu
>         Attachments: HDFS-10383.000.patch, HDFS-10383.001.patch, HDFS-10383.002.patch,
> There are a few of methods in {{DFSTestUtil}} that do not close the resource safely,
or elegantly. We can use the try-with-resource statement to address this problem.
> Specially, as {{DFSTestUtil}} is popularly used in test, we need to preserve any exceptions
thrown during the processing of the resource while still guaranteeing it's closed finally.
Take for example,the current implementation of {{DFSTestUtil#createFile()}} closes the FSDataOutputStream
in the {{finally}} block, and when closing if the internal {{DFSOutputStream#close()}} throws
any exception, which it often does, the exception thrown during the processing will be lost.
See this [test failure|https://builds.apache.org/job/PreCommit-HADOOP-Build/9320/testReport/org.apache.hadoop.hdfs/TestAsyncDFSRename/testAggressiveConcurrentAsyncRenameWithOverwrite/],
and we have to guess what was the root cause.
> Using try-with-resource, we can close the resources safely, and the exceptions thrown
both in processing and closing will be available (closing exception will be suppressed).

This message was sent by Atlassian JIRA

To unsubscribe, e-mail: hdfs-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-help@hadoop.apache.org

View raw message