hadoop-common-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Steve Loughran (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (HADOOP-13899) tune dynamodb client & tests; document
Date Tue, 13 Dec 2016 14:06:58 GMT

     [ https://issues.apache.org/jira/browse/HADOOP-13899?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel

Steve Loughran updated HADOOP-13899:
    Attachment: HADOOP-13899-HADOOP-13345-001.patch

Patch 001, to apply against the s3guard patch.

The changes mentioned, and

* {{DynamoDBMetadataStore}} to throw {{InterruptedIOException}} when IO operations/waits are
* {{DynamoDBClientFactory}} to use {{}}.getTrimmed()}} to get config options. (this should
be done throughout the module as general best practise)
* Move verifyFileStatus methods to S3ATestUtils for reuse; include {{status.toString()}} on
assertion message.

The patch to {{DynamoDBMetadataStore.initialize()}} to use .getBucket() to determine the bucket
is critical, the rest are improvements. 

I can see that the change to fail if the requested back end isn't available my be something
people disagree with, but consider this
* it avoids those support calls "S3 client work slower than promised"
* It avoids those support calls "data overwritten and lost on commits"

We don't want those, especially the latter.

> tune dynamodb client & tests; document
> --------------------------------------
>                 Key: HADOOP-13899
>                 URL: https://issues.apache.org/jira/browse/HADOOP-13899
>             Project: Hadoop Common
>          Issue Type: Sub-task
>          Components: fs/s3, test
>    Affects Versions: 3.0.0-alpha2
>            Reporter: Steve Loughran
>            Assignee: Steve Loughran
>         Attachments: HADOOP-13899-HADOOP-13345-001.patch
> While setting up clients for testing dynamo DB, make the tweaks to the dynamo store and
the s3guard code for better use downstream. These are the kind of things we need to round
off the code for production use.

This message was sent by Atlassian JIRA

To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org

View raw message