hadoop-common-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Sean Mackrory (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (HADOOP-14068) Add integration test version of TestMetadataStore for DynamoDB
Date Fri, 07 Jul 2017 21:51:00 GMT

     [ https://issues.apache.org/jira/browse/HADOOP-14068?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel

Sean Mackrory updated HADOOP-14068:
    Attachment: HADOOP-14068-HADOOP-13345.008.patch

I had been back to this, but after some previous refactoring it was more difficult to override
everything I needed to, and then some tests were failing for unusual reasons. I now have all
tests passing. Attaching a patch that does a few things:

- Does a bit of refactoring in TestDynamoDBMetadataStore so the parts that deal with the local
instance can be overridden, allowing the test to fall back on the integration test configuration.
- Provides an ITestDynamoDBMetadataStore class that overrides such things. It is not safe
to run concurrently with other tests - so unfortunately it added about 8 minutes to a test
run for me.
- Fixed the I/O reprovisioning test: previously, state changes happened immediately after
reprovisioning. Now, you have to wait for the table to return to the active state.
- Fixed testInitialization to not assume the bucket and table names match.

My main concern is the time it takes. I wouldn't be opposed to hiding this behind the -Dscale
option, although technically it's not a scale test. It is a *big* test, and is only useful
for catching bugs not caught against the Local version. Those have definitely existed before,
and developing this test helped me catch them, but I don't think the time is worth it for
every little commit.

> Add integration test version of TestMetadataStore for DynamoDB
> --------------------------------------------------------------
>                 Key: HADOOP-14068
>                 URL: https://issues.apache.org/jira/browse/HADOOP-14068
>             Project: Hadoop Common
>          Issue Type: Sub-task
>          Components: fs/s3
>            Reporter: Sean Mackrory
>            Assignee: Sean Mackrory
>         Attachments: HADOOP-14068-HADOOP-13345.001.patch, HADOOP-14068-HADOOP-13345.002.patch,
HADOOP-14068-HADOOP-13345.003.patch, HADOOP-14068-HADOOP-13345.004.patch, HADOOP-14068-HADOOP-13345.005.patch,
HADOOP-14068-HADOOP-13345.006.patch, HADOOP-14068-HADOOP-13345.007.patch, HADOOP-14068-HADOOP-13345.008.patch
> I tweaked TestDynamoDBMetadataStore to run against the actual Amazon DynamoDB service
(as opposed to the "local" edition). Several tests failed because of minor variations in behavior.
I think the differences that are clearly possible are enough to warrant extending that class
as an ITest (but obviously keeping the existing test so 99% of the the coverage remains even
when not configured for actual DynamoDB usage).

This message was sent by Atlassian JIRA

To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org

View raw message