hadoop-common-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Steve Loughran (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HADOOP-14553) Add (parallelized) integration tests to hadoop-azure
Date Fri, 30 Jun 2017 19:30:00 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-14553?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16070617#comment-16070617
] 

Steve Loughran commented on HADOOP-14553:
-----------------------------------------

Patch 004

# parallelise some more with better isolation
# Clean up some of the tests with: try with resources, use of ContracTestUtil asserts about
the state of the FS.

Metric of work, the non-scale IT tests run in under 3 minutes
{code}
 T E S T S
-------------------------------------------------------
Running org.apache.hadoop.fs.azure.contract.ITestAzureNativeContractMkdir
Running org.apache.hadoop.fs.azure.ITestAzureConcurrentOutOfBandIoWithSecureMode
Running org.apache.hadoop.fs.azure.integration.ITestFileSystemOperationExceptionMessage
Running org.apache.hadoop.fs.azure.contract.ITestAzureNativeContractAppend
Running org.apache.hadoop.fs.azure.contract.ITestAzureNativeContractCreate
Running org.apache.hadoop.fs.azure.contract.ITestAzureNativeContractSeek
Running org.apache.hadoop.fs.azure.contract.ITestAzureNativeContractOpen
Running org.apache.hadoop.fs.azure.ITestAzureConcurrentOutOfBandIo
Running org.apache.hadoop.fs.azure.contract.ITestAzureNativeContractRename
Running org.apache.hadoop.fs.azure.contract.ITestAzureNativeContractDelete
Running org.apache.hadoop.fs.azure.integration.ITestAzureHugeFiles
Running org.apache.hadoop.fs.azure.contract.ITestAzureNativeContractGetFileStatus
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 19.643 sec - in org.apache.hadoop.fs.azure.integration.ITestFileSystemOperationExceptionMessage
Running org.apache.hadoop.fs.azure.ITestNativeAzureFileSystemAppend
Tests run: 5, Failures: 0, Errors: 0, Skipped: 1, Time elapsed: 22.088 sec - in org.apache.hadoop.fs.azure.contract.ITestAzureNativeContractAppend
Running org.apache.hadoop.fs.azure.ITestNativeAzureFileSystemClientLogging
Tests run: 6, Failures: 0, Errors: 0, Skipped: 6, Time elapsed: 22.189 sec - in org.apache.hadoop.fs.azure.integration.ITestAzureHugeFiles
Running org.apache.hadoop.fs.azure.ITestNativeAzureFileSystemContractEmulator
Tests run: 43, Failures: 0, Errors: 0, Skipped: 43, Time elapsed: 1.068 sec - in org.apache.hadoop.fs.azure.ITestNativeAzureFileSystemContractEmulator
Running org.apache.hadoop.fs.azure.ITestNativeAzureFileSystemContractLive
Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 25.535 sec - in org.apache.hadoop.fs.azure.contract.ITestAzureNativeContractOpen
Running org.apache.hadoop.fs.azure.ITestNativeAzureFileSystemContractMocked
Tests run: 43, Failures: 0, Errors: 0, Skipped: 5, Time elapsed: 2.207 sec - in org.apache.hadoop.fs.azure.ITestNativeAzureFileSystemContractMocked
Running org.apache.hadoop.fs.azure.ITestNativeAzureFileSystemContractPageBlobLive
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 28.855 sec - in org.apache.hadoop.fs.azure.ITestAzureConcurrentOutOfBandIo
Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 7.192 sec - in org.apache.hadoop.fs.azure.ITestNativeAzureFileSystemClientLogging
Running org.apache.hadoop.fs.azure.ITestNativeAzureFileSystemMocked
Running org.apache.hadoop.fs.azure.ITestReadAndSeekPageBlobAfterWrite
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 43.493 sec - in org.apache.hadoop.fs.azure.ITestAzureConcurrentOutOfBandIoWithSecureMode
Tests run: 46, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 18.95 sec - in org.apache.hadoop.fs.azure.ITestNativeAzureFileSystemMocked
Tests run: 8, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 58.926 sec - in org.apache.hadoop.fs.azure.contract.ITestAzureNativeContractDelete
Tests run: 5, Failures: 0, Errors: 0, Skipped: 5, Time elapsed: 30.771 sec - in org.apache.hadoop.fs.azure.ITestReadAndSeekPageBlobAfterWrite
Tests run: 11, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 62.445 sec - in org.apache.hadoop.fs.azure.contract.ITestAzureNativeContractCreate
Tests run: 7, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 68.468 sec - in org.apache.hadoop.fs.azure.contract.ITestAzureNativeContractMkdir
Tests run: 8, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 70.282 sec - in org.apache.hadoop.fs.azure.contract.ITestAzureNativeContractRename
Tests run: 18, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 82.534 sec - in org.apache.hadoop.fs.azure.contract.ITestAzureNativeContractSeek
Tests run: 18, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 94.839 sec - in org.apache.hadoop.fs.azure.contract.ITestAzureNativeContractGetFileStatus
Tests run: 43, Failures: 0, Errors: 0, Skipped: 5, Time elapsed: 87.913 sec - in org.apache.hadoop.fs.azure.ITestNativeAzureFileSystemContractPageBlobLive
Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 101.172 sec - in org.apache.hadoop.fs.azure.ITestNativeAzureFileSystemAppend
Tests run: 43, Failures: 0, Errors: 0, Skipped: 5, Time elapsed: 106.015 sec - in org.apache.hadoop.fs.azure.ITestNativeAzureFileSystemContractLive

Results :

Tests run: 321, Failures: 0, Errors: 0, Skipped: 70

[INFO] 
[INFO] --- maven-failsafe-plugin:2.17:integration-test (sequential-integration-tests) @ hadoop-
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 02:20 min
[INFO] Finished at: 2017-06-30T20:02:21+01:00
[INFO] Final Memory: 35M/822M
[INFO] ------------------------------------------------------------------------
{code}

The big troublespot in migration and performance is the subclasses of {{NativeAzureFileSystemBaseTest}}.
These are trouble as some methods need hardcode paths; and the overall suites are long. They
are a bottleneck.

The solution here will not be to try and migrate the entire suite, but instead split it up
into sections for ease of migration and maintenance, *and* to allow the tests to run in parallel,
rather than just sequentially.

# group by function
# explicit root-path relative tests to be in their own abstract test suite, whose implementation
classes will run sequentially
# move incrementally.

> Add (parallelized) integration tests to hadoop-azure
> ----------------------------------------------------
>
>                 Key: HADOOP-14553
>                 URL: https://issues.apache.org/jira/browse/HADOOP-14553
>             Project: Hadoop Common
>          Issue Type: Sub-task
>          Components: fs/azure
>    Affects Versions: 2.9.0
>            Reporter: Steve Loughran
>            Assignee: Steve Loughran
>         Attachments: HADOOP-14553-001.patch, HADOOP-14553-002.patch, HADOOP-14553-003.patch,
HADOOP-14553-004.patch
>
>
> The Azure tests are slow to run as they are serialized, as they are all called Test*
there's no clear differentiation from unit tests which Jenkins can run, and integration tests
which it can't.
> Move the azure tests {{Test*}} to integration tests {{ITest*}}, parallelize (which includes
having separate paths for every test suite). The code in hadoop-aws's POM &c show what
to do.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org


Mime
View raw message