hadoop-common-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Chris Nauroth (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (HADOOP-13113) Enable parallel test execution for hadoop-aws.
Date Sat, 07 May 2016 06:57:12 GMT

     [ https://issues.apache.org/jira/browse/HADOOP-13113?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel

Chris Nauroth updated HADOOP-13113:
    Attachment: HADOOP-13113.001.patch

I'm attaching patch v001.  This uses the same parallel test execution techniques that have
been effective in hadoop-common and hadoop-hdfs.  I updated pom.xml to add the {{parallel-tests}}
profile and changed some test code to generate testing paths based on the Surefire fork number,
so that they're guaranteed to be isolated from each other.  HADOOP-12801 is a pre-requisite
for this patch.

There are some limitations to this approach.  Tests that cover root directory access, multi-part
upload, or YARN usage would run into conflicts on shared resources.  To handle this, I've
moved those tests to a separate Surefire execution step, which runs sequentially instead of
parallelized.  A future improvement could be to refactor tests related to root out of {{FileSystemContractBaseTest}},
so that the non-root tests in that suite can run in parallel.  {{TestS3AFileSystemContract}}
takes ~90 seconds to run, so it would be nice to parallelize that with other suites.  I chose
not to step outside hadoop-aws for the scope of the current patch though.

While I was editing pom.xml, I also removed an unused dependency on Jackson.  This change
is backwards-compatible, because clients who had been relying on receiving Jackson via transitive
dependency through hadoop-aws will continue to receive it through the dependency chain hadoop-aws
-> hadoop-common -> Jackson.

In my environment, plain {{mvn test}} runs in ~30 minutes, and {{mvn -Pparallel-tests test}}
runs in ~13 minutes.

[~stevel@apache.org], would you please review when you get a chance?

> Enable parallel test execution for hadoop-aws.
> ----------------------------------------------
>                 Key: HADOOP-13113
>                 URL: https://issues.apache.org/jira/browse/HADOOP-13113
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: test
>            Reporter: Chris Nauroth
>            Assignee: Chris Nauroth
>         Attachments: HADOOP-13113.001.patch
> The full hadoop-aws test suite takes ~30 minutes to execute.  The tests spend most of
their time blocked on network I/O with the S3 back-end, but they don't saturate the bandwidth
of the NIC.  We can improve overall execution time by enabling parallel test execution.

This message was sent by Atlassian JIRA

To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org

View raw message