hadoop-common-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Aaron Fabbri (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HADOOP-13904) DynamoDBMetadataStore to handle DDB throttling failures through retry policy
Date Wed, 01 Feb 2017 01:27:52 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-13904?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15847856#comment-15847856
] 

Aaron Fabbri commented on HADOOP-13904:
---------------------------------------

Thank you for the helpful review [~stevel@apache.org]

{quote}
do these run iff `-Dscale` is set? 
{quote}

Yep.. They extend {{S3AScaleTestBase}}.

{quote}
clearMetadataStore(ms, count); may need to go into a finally clause of testMoves()
{quote}

Good call.  I'll do the same in testPut() as well.  I'll retest with those changes and get
you timing details as well.

These could run in parallel if we ensured they had their own DDB table to work with.  I think
it would be more simple just to mark them as serial for scale tests.  I'll add that as well.








> DynamoDBMetadataStore to handle DDB throttling failures through retry policy
> ----------------------------------------------------------------------------
>
>                 Key: HADOOP-13904
>                 URL: https://issues.apache.org/jira/browse/HADOOP-13904
>             Project: Hadoop Common
>          Issue Type: Sub-task
>          Components: fs/s3
>    Affects Versions: HADOOP-13345
>            Reporter: Steve Loughran
>            Assignee: Aaron Fabbri
>         Attachments: HADOOP-13904-HADOOP-13345.001.patch
>
>
> When you overload DDB, you get error messages warning of throttling, [as documented by
AWS|http://docs.aws.amazon.com/amazondynamodb/latest/developerguide/Programming.Errors.html#Programming.Errors.MessagesAndCodes]
> Reduce load on DDB by doing a table lookup before the create, then, in table create/delete
operations and in get/put actions, recognise the error codes and retry using an appropriate
retry policy (exponential backoff + ultimate failure) 



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org


Mime
View raw message