hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Mingliang Liu (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (HDFS-10383) Safely close resources in DFSTestUtil
Date Thu, 12 May 2016 01:07:12 GMT

     [ https://issues.apache.org/jira/browse/HDFS-10383?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Mingliang Liu updated HDFS-10383:
---------------------------------
    Priority: Major  (was: Minor)

The v1 patch revisited the {{DFSTestUtil}} class.

# {{urlGetBytes}} is still using the {{IOUtils.copyBytes(conn.getInputStream(), out, 4096,
true);}} as the {{URLConnection}} is not implementing {{AutoCloseable}}. We need to close
its input stream explicitly and the {{IOUtils}} works just fine.
# {{waitCorruptReplicas}} swallows all the exception thrown by either processing the resource
or closing the resource. Ignored.
# {{runOperation}} runs a set of operations and generate all edit logs, and thus explicit
closing operation makes the edit log history clear for test and debugging. Ignored.

Other than these, all other places have been switched to try-with-resource statement.

> Safely close resources in DFSTestUtil
> -------------------------------------
>
>                 Key: HDFS-10383
>                 URL: https://issues.apache.org/jira/browse/HDFS-10383
>             Project: Hadoop HDFS
>          Issue Type: Bug
>          Components: test
>            Reporter: Mingliang Liu
>            Assignee: Mingliang Liu
>         Attachments: HDFS-10383.000.patch
>
>
> There are a few of methods in {{DFSTestUtil}} that do not close the resource safely,
or elegantly. We can use the try-with-resource statement to address this problem.
> Specially, as {{DFSTestUtil}} is popularly used in test, we need to preserve any exceptions
thrown during the processing of the resource while still guaranteeing it's closed finally.
Take for example,the current implementation of {{DFSTestUtil#createFile()}} closes the FSDataOutputStream
in the {{finally}} block, and when closing if the internal {{DFSOutputStream#close()}} throws
any exception, which it often does, the exception thrown during the processing will be lost.
See this [test failure|https://builds.apache.org/job/PreCommit-HADOOP-Build/9320/testReport/org.apache.hadoop.hdfs/TestAsyncDFSRename/testAggressiveConcurrentAsyncRenameWithOverwrite/],
and we have to guess what was the root cause.
> Using try-with-resource, we can close the resources safely, and the exceptions thrown
both in processing and closing will be available (closing exception will be suppressed).



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: hdfs-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-help@hadoop.apache.org


Mime
View raw message