Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
Date: Fri, 1 Sep 2017 13:56:00 +0000 (UTC)
From: "Steve Loughran (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <JIRA.13030263.1482451989000.9109.1504274160552@Atlassian.JIRA>
In-Reply-To: <JIRA.13030263.1482451989000@Atlassian.JIRA>
References: <JIRA.13030263.1482451989000@Atlassian.JIRA> <JIRA.13030263.1482451989664@jira-lw-us.apache.org>
Subject: [jira] [Commented] (HADOOP-13936) S3Guard: DynamoDB can go out of
 sync with S3AFileSystem::delete operation
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
archived-at: Fri, 01 Sep 2017 13:56:05 -0000


    [ https://issues.apache.org/jira/browse/HADOOP-13936?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16150564#comment-16150564 ] 

Steve Loughran commented on HADOOP-13936:
-----------------------------------------

what to do?

# improve prune
# do it in the proposed fsck?

> S3Guard: DynamoDB can go out of sync with S3AFileSystem::delete operation
> -------------------------------------------------------------------------
>
>                 Key: HADOOP-13936
>                 URL: https://issues.apache.org/jira/browse/HADOOP-13936
>             Project: Hadoop Common
>          Issue Type: Sub-task
>          Components: fs/s3
>    Affects Versions: 3.0.0-beta1
>            Reporter: Rajesh Balamohan
>            Priority: Minor
>
> As a part of {{S3AFileSystem.delete}} operation {{innerDelete}} is invoked, which deletes keys from S3 in batches (default is 1000). But DynamoDB is updated only at the end of this operation. This can cause issues when deleting large number of keys. 
> E.g, it is possible to get exception after deleting 1000 keys and in such cases dynamoDB would not be updated. This can cause DynamoDB to go out of sync. 


--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org