hadoop-common-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Chris Nauroth (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (HADOOP-13459) hadoop-aws runs several test cases repeatedly, causing unnecessarily long running time.
Date Wed, 03 Aug 2016 00:08:20 GMT

     [ https://issues.apache.org/jira/browse/HADOOP-13459?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Chris Nauroth updated HADOOP-13459:
-----------------------------------
    Attachment: HADOOP-13459.001.patch

This patch refactors the tests to introduce a new abstract class: {{AbstractWasbTestBase}}.
 The vast majority of test suites can subclass this to get just the setup and teardown logic
without accidentally re-running test cases.

Here is some information on running time for the impacted test suites before and after the
patch.

*Before:*
Running org.apache.hadoop.fs.azure.TestFileSystemOperationExceptionHandling
Tests run: 62, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 325.178 sec - in org.apache.hadoop.fs.azure.TestFileSystemOperationExceptionHandling
Running org.apache.hadoop.fs.azure.TestFileSystemOperationsExceptionHandlingMultiThreaded
Tests run: 58, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 316.563 sec - in org.apache.hadoop.fs.azure.TestFileSystemOperationsExceptionHandlingMultiThreaded
Running org.apache.hadoop.fs.azure.TestNativeAzureFileSystemAppend
Tests run: 52, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 392.583 sec - in org.apache.hadoop.fs.azure.TestNativeAzureFileSystemAppend
Running org.apache.hadoop.fs.azure.TestNativeAzureFileSystemAtomicRenameDirList
Tests run: 47, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 284.985 sec - in org.apache.hadoop.fs.azure.TestNativeAzureFileSystemAtomicRenameDirList
Running org.apache.hadoop.fs.azure.TestNativeAzureFileSystemClientLogging
Tests run: 48, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 300.001 sec - in org.apache.hadoop.fs.azure.TestNativeAzureFileSystemClientLogging
Running org.apache.hadoop.fs.azure.TestNativeAzureFileSystemLive
Tests run: 51, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 296.564 sec - in org.apache.hadoop.fs.azure.TestNativeAzureFileSystemLive
Running org.apache.hadoop.fs.azure.TestNativeAzureFileSystemMocked
Tests run: 46, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 16.994 sec - in org.apache.hadoop.fs.azure.TestNativeAzureFileSystemMocked

*After:*
Running org.apache.hadoop.fs.azure.TestFileSystemOperationExceptionHandling
Tests run: 16, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 30.088 sec - in org.apache.hadoop.fs.azure.TestFileSystemOperationExceptionHandling
Running org.apache.hadoop.fs.azure.TestFileSystemOperationsExceptionHandlingMultiThreaded
Tests run: 12, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 25.198 sec - in org.apache.hadoop.fs.azure.TestFileSystemOperationsExceptionHandlingMultiThreaded
Running org.apache.hadoop.fs.azure.TestNativeAzureFileSystemAppend
Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 76.855 sec - in org.apache.hadoop.fs.azure.TestNativeAzureFileSystemAppend
Running org.apache.hadoop.fs.azure.TestNativeAzureFileSystemAtomicRenameDirList
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1.6 sec - in org.apache.hadoop.fs.azure.TestNativeAzureFileSystemAtomicRenameDirList
Running org.apache.hadoop.fs.azure.TestNativeAzureFileSystemClientLogging
Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1.944 sec - in org.apache.hadoop.fs.azure.TestNativeAzureFileSystemClientLogging
Running org.apache.hadoop.fs.azure.TestNativeAzureFileSystemLive
Tests run: 51, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 307.217 sec - in org.apache.hadoop.fs.azure.TestNativeAzureFileSystemLive
Running org.apache.hadoop.fs.azure.TestNativeAzureFileSystemMocked
Tests run: 46, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 16.915 sec - in org.apache.hadoop.fs.azure.TestNativeAzureFileSystemMocked


> hadoop-aws runs several test cases repeatedly, causing unnecessarily long running time.
> ---------------------------------------------------------------------------------------
>
>                 Key: HADOOP-13459
>                 URL: https://issues.apache.org/jira/browse/HADOOP-13459
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: fs/azure, test
>            Reporter: Chris Nauroth
>            Assignee: Chris Nauroth
>            Priority: Minor
>         Attachments: HADOOP-13459.001.patch
>
>
> Within hadoop-azure, we have the {{NativeAzureFileSystemBaseTest}} abstract class, which
defines setup and teardown to handle the Azure storage account and also defines multiple test
cases.  This class originally was contributed to provide a layer of indirection for running
the same test cases in live mode or mock mode: {{TestNativeAzureFileSystemLive}} and {{TestNativeAzureFileSystemMocked}}.
 It appears that since then, we created multiple new test suites that subclassed {{NativeAzureFileSystemBaseTest}}
for the benefit of getting the common setup and teardown code, but also with the side effect
of running the inherited test cases repeatedly.  This is a significant factor in the overall
execution time of the hadoop-azure tests.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org


Mime
View raw message