hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Vinayakumar B (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HDFS-10256) Use GenericTestUtils.getTestDir method in tests for temporary directories
Date Wed, 20 Apr 2016 06:09:25 GMT

    [ https://issues.apache.org/jira/browse/HDFS-10256?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15249340#comment-15249340
] 

Vinayakumar B commented on HDFS-10256:
--------------------------------------

bq. MiniDFSCluster#shutdown() registers base_dir to be deleted on shutdown. If this gets slow,
the next test JVM will start to run before the shutdown hook completes. But forcing every
test to call shutdown(true) can slowdown things. Instead, each instance should get a random
base_dir, so that the deletion through shutdown hook and the subsequent new test setup can
overlap.
I have investigated a bit on this. Registering a dir to be deleted during shutdown will not
actually delete the directory unless its empty. I have tried to register all files/directories
recursively to be deleted on shutdown. But this will sometimes keeps the empty directories.
Instead, I have used unique base directory (testclassname.methodname) for each MiniDFSCluster
instance and FileUtils.fullyDelete()}} during shutdown itself, instead of registering for
deleteOnExit() in MiniDFSCluster#shutdown().
All tests which actually uses MiniDfsCluster will delete its basedir. Apart from this, there
are some more tests, which creates directories on its own ( using some Random String), and
these may not get deleted after test run.

At the end of test run, there was ~300MB of data created in "hadoop-hdfs/target/test/data"
directory, which is way less than ~6GB reported by Chris.


So I am not sure why {{base_dir.deleteOnExit()}} was used during {{MiniDfsCluster#shutdown()}},
which will not actually delete anything because base_dir will not be empty.

> Use GenericTestUtils.getTestDir method in tests for temporary directories
> -------------------------------------------------------------------------
>
>                 Key: HDFS-10256
>                 URL: https://issues.apache.org/jira/browse/HDFS-10256
>             Project: Hadoop HDFS
>          Issue Type: Improvement
>          Components: build, test
>            Reporter: Vinayakumar B
>            Assignee: Vinayakumar B
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message