hadoop-common-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Steve Loughran (JIRA)" <j...@apache.org>
Subject [jira] [Resolved] (HADOOP-15563) S3Guard to support creating on-demand DDB tables
Date Fri, 07 Jun 2019 17:31:00 GMT

     [ https://issues.apache.org/jira/browse/HADOOP-15563?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel

Steve Loughran resolved HADOOP-15563.
       Resolution: Fixed
    Fix Version/s: 3.3.0
     Release Note: S3Guard now defaults to creating DynamoDB tables as "On-Demand", rather
than with a prepaid IO capacity. This reduces costs when idle to only the storage of the metadata
entries, while delivering significantly faster performance during query planning and other
bursts of IO. Consult the S3Guard documentation for further details.

committed to trunk after a +1 from sean on the PR. Thanks!

> S3Guard to support creating on-demand DDB tables
> ------------------------------------------------
>                 Key: HADOOP-15563
>                 URL: https://issues.apache.org/jira/browse/HADOOP-15563
>             Project: Hadoop Common
>          Issue Type: Sub-task
>          Components: fs/s3
>    Affects Versions: 3.1.0
>            Reporter: Steve Loughran
>            Assignee: Steve Loughran
>            Priority: Major
>             Fix For: 3.3.0
> To keep costs down on DDB, autoscaling is a key feature: you set the max values and when
idle, you don't get billed, *at the cost of delayed scale time and risk of not getting the
max value when AWS is busy*
> It can be done from the AWS web UI, but not in the s3guard init and set-capacity calls
> It can be done [through the API|https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/AutoScaling.HowTo.SDK.html]
> Usual issues then: wiring up, CLI params, testing. It'll be hard to test.
> Fully support On-demand DDB tables in S3Guard
> * create (0, 0) will create an on-demand table.
> * set capacity (0, 0) will create an on-demand table.
> * once a table is on demand, any set capacity command other than to (0, 0) will then
> * when loading table, note if it is on-demand or not
> * if on demand, prune() doesn't bother to throttle requests any more by sleeping.

This message was sent by Atlassian JIRA

To unsubscribe, e-mail: common-dev-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-dev-help@hadoop.apache.org

View raw message