hadoop-common-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Steve Loughran (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HADOOP-13021) Hadoop swift driver unit test should use unique directory for each run
Date Wed, 13 Apr 2016 09:54:25 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-13021?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15238978#comment-15238978

Steve Loughran commented on HADOOP-13021:

You shouldn't be editing core-site.xml: if that is where you are putting login details for
an object store, and you've committed that to some form of SCM: stop, revert, change the login

The JUnit tests in swift are designed to pick up details in auth-keys.xml. Not only can you
have different ones for each machine, if you have an absolute XML include reference you can
pick up hard values. And you can then set up a separate s3 bucket for each machine.

I would recommend you isolate with a separate bucket per host, not path underneath. things
like test cleanup may interfere ... it's designed to purge everything to ensure you don't
run up storage bills.

  <include xmlns="http://www.w3.org/2001/XInclude"
    href="file:///users/stevel/.ssh/auth-keys.xml" />

> Hadoop swift driver unit test should use unique directory for each run
> ----------------------------------------------------------------------
>                 Key: HADOOP-13021
>                 URL: https://issues.apache.org/jira/browse/HADOOP-13021
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: fs/swift
>    Affects Versions: 2.7.2
>            Reporter: Chen He
>            Assignee: Chen He
>              Labels: unit-test
> Since all "unit test" in swift package are actually functionality test, it requires server's
information in the core-site.xml file. However, multiple unit test runs on difference machines
using the same core-site.xml file will result in some unit tests failure. For example:
> In TestSwiftFileSystemBasicOps.java
> public void testMkDir() throws Throwable {
>     Path path = new Path("/test/MkDir");
>     fs.mkdirs(path);
>     //success then -so try a recursive operation
>     fs.delete(path, true);
>   }
> It is possible that machine A and B are running "mvn clean install" using same core-site.xml
file. However, machine A run testMkDir() first and delete the dir, but machine B just tried
to run fs.delete(path,true). It will report failure. This is just an example. There are many
similar cases in the unit test sets. I would propose we use a unique dir for each unit test
run instead of using "Path path = new Path("/test/MkDir")" for all concurrent runs

This message was sent by Atlassian JIRA

View raw message