hadoop-common-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Chris Nauroth (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HADOOP-13231) Isolate test path used by a few S3A tests for more reliable parallel execution.
Date Mon, 13 Jun 2016 16:41:21 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-13231?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15327714#comment-15327714

Chris Nauroth commented on HADOOP-13231:

bq. Should those named tests be .class rather than .java?

Using .java is correct, as per Maven's documentation on [Inclusions and Exclusions of Tests|http://maven.apache.org/surefire/maven-surefire-plugin/examples/inclusion-exclusion.html].

Repeating a comment from HADOOP-13139:

mvn test -Dtest=TestS3A*  -Pparallel-tests -DtestsThreadCount=2

[~stevel@apache.org], when you pass a {{-Dtest=}} argument, you are overriding the {{<includes>}}
list passed to Surefire during the build.  Passing {{TestS3A*}} matches all the S3A test suites,
including the ones I listed in {{<excludes>}} to prevent parallel execution.  I just
tested this, and it looks like if Surefire finds the same test matching both the whitelist
and the blacklist, then the whitelist wins.  Therefore, tests that can't handle parallel execution
end up running in the parallel phase.

Unfortunately, I think this means you can't reliably do both {{-Pparallel-tests}} and {{-Dtest=TestS3A*}}
in the same {{mvn}} run.  I had flagged all multi-part upload tests to run outside the parallel
phase because of the abort-existing-uploads-on-init logic, so I wouldn't expect them to work
well in the parallel phase.

It would be nice to be able to run only S3A tests in parallel for more rapid iterations on
patches that touch only S3A code.  I think we'd need to use some other naming convention on
all non-parallel-safe test suite classes to support that, so that you could specify a wildcard
to {{-Dtest=}} that wouldn't match those suites.

> Isolate test path used by a few S3A tests for more reliable parallel execution.
> -------------------------------------------------------------------------------
>                 Key: HADOOP-13231
>                 URL: https://issues.apache.org/jira/browse/HADOOP-13231
>             Project: Hadoop Common
>          Issue Type: Sub-task
>          Components: fs/s3, test
>            Reporter: Chris Nauroth
>            Assignee: Chris Nauroth
>            Priority: Minor
> I have noticed a few more spots in S3A tests that do not make use of the isolated test
directory path when running in parallel mode.  While I don't have any evidence that this is
really causing problems for parallel test runs right now, it would still be good practice
to clean these up.

This message was sent by Atlassian JIRA

To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org

View raw message