hadoop-common-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Steve Loughran (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HADOOP-13936) S3Guard: DynamoDB can go out of sync with S3AFileSystem::delete operation
Date Fri, 06 Jan 2017 19:28:58 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-13936?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15805379#comment-15805379
] 

Steve Loughran commented on HADOOP-13936:
-----------------------------------------

Smaller batch sizes? It could cause throttling problems

> S3Guard: DynamoDB can go out of sync with S3AFileSystem::delete operation
> -------------------------------------------------------------------------
>
>                 Key: HADOOP-13936
>                 URL: https://issues.apache.org/jira/browse/HADOOP-13936
>             Project: Hadoop Common
>          Issue Type: Sub-task
>          Components: fs/s3
>            Reporter: Rajesh Balamohan
>            Priority: Minor
>
> As a part of {{S3AFileSystem.delete}} operation {{innerDelete}} is invoked, which deletes
keys from S3 in batches (default is 1000). But DynamoDB is updated only at the end of this
operation. This can cause issues when deleting large number of keys. 
> E.g, it is possible to get exception after deleting 1000 keys and in such cases dynamoDB
would not be updated. This can cause DynamoDB to go out of sync. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org


Mime
View raw message