hadoop-common-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Chris Nauroth (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (HADOOP-13446) S3Guard: Support running isolated unit tests separate from AWS integration tests.
Date Thu, 04 Aug 2016 05:49:20 GMT

     [ https://issues.apache.org/jira/browse/HADOOP-13446?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel

Chris Nauroth updated HADOOP-13446:
    Attachment: HADOOP-13446-HADOOP-13345.001.patch

I'm attaching patch v001, implementing what I had in mind.  Summary:

# Add maven-failsafe-plugin to the build.
# Rename all test classes that require connecting to S3 from Test*.java to ITest*.java to
fit the maven-failsafe-plugin naming convention.  These will run during the Maven integration-test
phase instead of the test phase.
# For test classes that are true unit tests wihtout any dependency on S3, leave them untouched.
 These will continue to use the maven-surefire-plugin naming convention and run during the
test phase.  There are 5 existing test suite classes like this already, and I expect S3Guard
will add more.

The patch file is large, but it's mostly the result of scripting a bunch of renames.  The
Java code in the test classes is unchanged, so I don't expect anyone to do a line-by-line
review of the test suites.

Due to Maven quirks, the execution definition for maven-failsafe-plugin when running in sequential
mode must not go into the main build definition.  Otherwise, it would override the parallel
definition.  The workaround for this is to put the definition for sequential mode behind its
own sequential-tests profile, triggered only when the parallel-tests profile is off.  This
requires use of activation by presence (or lack of presence) of a property, not a profile
name.  For this to work properly, we are now going to have to specify {{-Dparallel-tests}}
instead of {{-Pparallel-tests}} if we want a parallel run.  (Usage when running in sequential
mode is unchanged.)

Considering use of the property and use of the new Maven lifecycle phases, that means we support
the following usage:

*{{mvn test}}* - Run only the unit tests.  No connection to S3 is required.  These tests run
even if S3 credentials are not present in the environment.
*{{mvn verify}}* - Run unit tests and integration tests.  If S3 credentials are not present,
then the integration tests are skipped automatically.
*{{mvn -Dparallel-tests test}}* - Run only the unit tests, in parallel.
*{{mvn -Dparallel-tests verify}}* - Run unit tests and integration tests, in parallel.  If
S3 credentials are not present, then the integration tests are skipped automatically.

Our policy now should be that every patch must demonstrate a clean {{mvn verify}} run before

I recommend not running {{mvn -Pparallel-tests verify}} ({{-P}} instead of {{-D}}).  Due to
the Maven quirks described above, you won't actually get parallel execution.

I have tested all of the above scenarios.  Running a specific test with {{-Dtest=}} works

> S3Guard: Support running isolated unit tests separate from AWS integration tests.
> ---------------------------------------------------------------------------------
>                 Key: HADOOP-13446
>                 URL: https://issues.apache.org/jira/browse/HADOOP-13446
>             Project: Hadoop Common
>          Issue Type: Sub-task
>          Components: fs/s3
>            Reporter: Chris Nauroth
>            Assignee: Chris Nauroth
>         Attachments: HADOOP-13446-HADOOP-13345.001.patch
> Currently, the hadoop-aws module only runs Surefire if AWS credentials have been configured.
 This implies that all tests must run integrated with the AWS back-end.  It also means that
no tests run as part of ASF pre-commit.  This issue proposes for the hadoop-aws module to
support running isolated unit tests without integrating with AWS.  This will benefit S3Guard,
because we expect the need for isolated mock-based testing to simulate eventual consistency
behavior.  It also benefits hadoop-aws in general by allowing pre-commit to do something more

This message was sent by Atlassian JIRA

To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org

View raw message