hadoop-common-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Steve Loughran (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HADOOP-14553) Add (parallelized) integration tests to hadoop-azure
Date Fri, 23 Jun 2017 21:14:00 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-14553?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16061505#comment-16061505

Steve Loughran commented on HADOOP-14553:

patch 002
There's some careful parallelization of a few tests, enough to show that it does work if you
do it case by case, the use of shared/hard coded paths being the enemy. I've moved over all
the contract tests as we know they parallize well once you set up the contract to return fork-specific

I've uprated a limited set  oftests so far, enough to show things work and to achieve measurable
speeups already. Here is a test where the total time to run these tests is 3 minutes, rather
than the 5 minutes it would take otherwise. Tagging the huge files and read and seek as scale
tests means that non-scale test runs would be even faster.


* all the contract tests in {{org.apache.hadoop.fs.azure.contract}}, tweaking {{NativeAzureFileSystemContract}}
to use the fork ID sysprop to generate unique paths.
* {{ITestFileSystemOperationExceptionMessage}}. This was taking so long as it was needlessly
subclassing {{NativeAzureFileSystemBaseTest}} and its 50 tests. Moving the single method to
a new IT test fixed that.
* {{ITestNativeAzureFileSystemContractLive}}: again, known to parallelize.
* {{ITestReadAndSeekPageBlobAfterWrite}} self-contained, slow (it's been tagged as a scale
test), and validates the ability for pageblob paths to be per-fork too.

* docs in `test_azure.md`


> mvit   -Dparallel-tests -DtestsThreadCount=8 -Dscale -Dtest=none

 T E S T S
Running org.apache.hadoop.fs.azure.contract.ITestAzureNativeContractAppend
Running org.apache.hadoop.fs.azure.contract.ITestAzureNativeContractGetFileStatus
Running org.apache.hadoop.fs.azure.contract.ITestAzureNativeContractCreate
Running org.apache.hadoop.fs.azure.contract.ITestAzureNativeContractMkdir
Running org.apache.hadoop.fs.azure.contract.ITestAzureNativeContractRename
Running org.apache.hadoop.fs.azure.contract.ITestAzureNativeContractDistCp
Running org.apache.hadoop.fs.azure.contract.ITestAzureNativeContractDelete
Running org.apache.hadoop.fs.azure.contract.ITestAzureNativeContractOpen
Tests run: 5, Failures: 0, Errors: 0, Skipped: 1, Time elapsed: 11.567 sec - in org.apache.hadoop.fs.azure.contract.ITestAzureNativeContractAppend
Running org.apache.hadoop.fs.azure.contract.ITestAzureNativeContractSeek
Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 12.605 sec - in org.apache.hadoop.fs.azure.contract.ITestAzureNativeContractOpen
Running org.apache.hadoop.fs.azure.integration.ITestAzureHugeFiles
Tests run: 8, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 19.765 sec - in org.apache.hadoop.fs.azure.contract.ITestAzureNativeContractDelete
Running org.apache.hadoop.fs.azure.integration.ITestFileSystemOperationExceptionMessage
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1.948 sec - in org.apache.hadoop.fs.azure.integration.ITestFileSystemOperationExceptionMessage
Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 22.087 sec - in org.apache.hadoop.fs.azure.contract.ITestAzureNativeContractRename
Running org.apache.hadoop.fs.azure.ITestNativeAzureFileSystemContractLive
Running org.apache.hadoop.fs.azure.ITestReadAndSeekPageBlobAfterWrite
Tests run: 11, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 23.702 sec - in org.apache.hadoop.fs.azure.contract.ITestAzureNativeContractCreate
Tests run: 7, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 43.769 sec - in org.apache.hadoop.fs.azure.contract.ITestAzureNativeContractMkdir
Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 34.872 sec - in org.apache.hadoop.fs.azure.integration.ITestAzureHugeFiles
Tests run: 18, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 62.163 sec - in org.apache.hadoop.fs.azure.contract.ITestAzureNativeContractSeek
Tests run: 18, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 79.653 sec - in org.apache.hadoop.fs.azure.contract.ITestAzureNativeContractGetFileStatus
Tests run: 43, Failures: 0, Errors: 0, Skipped: 5, Time elapsed: 98.776 sec - in org.apache.hadoop.fs.azure.ITestNativeAzureFileSystemContractLive
Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 150.19 sec - in org.apache.hadoop.fs.azure.contract.ITestAzureNativeContractDistCp
Tests run: 5, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 165.022 sec - in org.apache.hadoop.fs.azure.ITestReadAndSeekPageBlobAfterWrite

Results :

Tests run: 138, Failures: 0, Errors: 0, Skipped: 6

[INFO] ------------------------------------------------------------------------
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 03:14 min (Wall Clock)
[INFO] Finished at: 2017-06-23T21:52:01+01:00
[INFO] Final Memory: 38M/820M
[INFO] ------------------------------------------------------------------------

You need to run the full tests with `mvn verify`, for some reason they don't run on `mvn integration-test`,
whereas they do on hadoop-aws.

The main tests still take forever, for a loose definition of forever, they still leak wasb
storage containers. As well as tracking that down, I'm tempted to have an IT test in the sequential
phase which lists all matching containers and deletes them. For now, main issue is making
sure things don't time out. Of course, the more that get moved to parallel tests, the faster
the runs, the less risk of timeouts.

I don't plan to move any more tests over in this JIRA: there's enough to set up the parallel
test execution framework & explore its functionality, with the new seek test it's set
up for seek changes. The remaining tests would go if/when {{AbstractWasbTestBase}} was brought
up as an IT tests, then case-by-case as paths get isolated. The slow tests like {{TestAzureFileSystemInstrumentation}}
would be the first targets.

Also, I'd look really hard about whether it's worthwhile supporting the emulation. Life is
simpler if the policy is just "test against real Azure". Something for another day, but people
should think about it. A test run doesn't cost much and its more realistic; even long-haul
test runs like mine get to find interesting network errors.

Would be good for others to play with this, perhaps with some review by [~cnauroth].

> Add (parallelized) integration tests to hadoop-azure
> ----------------------------------------------------
>                 Key: HADOOP-14553
>                 URL: https://issues.apache.org/jira/browse/HADOOP-14553
>             Project: Hadoop Common
>          Issue Type: Sub-task
>          Components: fs/azure
>    Affects Versions: 2.9.0
>            Reporter: Steve Loughran
>            Assignee: Steve Loughran
>         Attachments: HADOOP-14553-001.patch, HADOOP-14553-002.patch
> The Azure tests are slow to run as they are serialized, as they are all called Test*
there's no clear differentiation from unit tests which Jenkins can run, and integration tests
which it can't.
> Move the azure tests {{Test*}} to integration tests {{ITest*}}, parallelize (which includes
having separate paths for every test suite). The code in hadoop-aws's POM &c show what
to do.

This message was sent by Atlassian JIRA

To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org

View raw message