hadoop-common-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Aaron Fabbri (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HADOOP-13904) DynamoDBMetadataStore to handle DDB throttling failures through retry policy
Date Wed, 15 Feb 2017 22:53:42 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-13904?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15868754#comment-15868754

Aaron Fabbri commented on HADOOP-13904:

Ok.. I started on this, again realizing what a crazy API choice it would have been for the
DDB SDK to treat this "nothing was processed" throttling behavior as completely separate from
the "some things were processed" throttling case.

Handling that would have been pretty complex, with the unprocessedItems only being used in
the former case and not the latter (i.e. if you get partially throttled, then fully throttled
on first retry, you have to retry the previous unprocessedItems list, etc.)

I digged deeper in the AWS SDK docs and found this:

ProvisionedThroughputExceededException - Your request rate is too high. The AWS SDKs for DynamoDB
automatically retry requests that receive this exception. 

This is consistent with my testing: Even with a tiny provisioned throughput and abusive scale
test, I never once saw this exception.  I did frequently see throttling via the UnprocessedItems
return value.

So, I now think we should *not* try to handle that exception here.  I'm trying to find the
SDK code to confirm this behavior.

> DynamoDBMetadataStore to handle DDB throttling failures through retry policy
> ----------------------------------------------------------------------------
>                 Key: HADOOP-13904
>                 URL: https://issues.apache.org/jira/browse/HADOOP-13904
>             Project: Hadoop Common
>          Issue Type: Sub-task
>          Components: fs/s3
>    Affects Versions: HADOOP-13345
>            Reporter: Steve Loughran
>            Assignee: Aaron Fabbri
>         Attachments: HADOOP-13904-HADOOP-13345.001.patch, HADOOP-13904-HADOOP-13345.002.patch
> When you overload DDB, you get error messages warning of throttling, [as documented by
> Reduce load on DDB by doing a table lookup before the create, then, in table create/delete
operations and in get/put actions, recognise the error codes and retry using an appropriate
retry policy (exponential backoff + ultimate failure) 

This message was sent by Atlassian JIRA

To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org

View raw message