Return-Path: X-Original-To: archive-asf-public-internal@cust-asf2.ponee.io Delivered-To: archive-asf-public-internal@cust-asf2.ponee.io Received: from cust-asf.ponee.io (cust-asf.ponee.io [163.172.22.183]) by cust-asf2.ponee.io (Postfix) with ESMTP id 567282004F5 for ; Fri, 1 Sep 2017 15:56:10 +0200 (CEST) Received: by cust-asf.ponee.io (Postfix) id 5303A16D107; Fri, 1 Sep 2017 13:56:10 +0000 (UTC) Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by cust-asf.ponee.io (Postfix) with SMTP id 99DD116D104 for ; Fri, 1 Sep 2017 15:56:09 +0200 (CEST) Received: (qmail 45443 invoked by uid 500); 1 Sep 2017 13:56:07 -0000 Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Delivered-To: mailing list common-issues@hadoop.apache.org Received: (qmail 45431 invoked by uid 99); 1 Sep 2017 13:56:07 -0000 Received: from pnap-us-west-generic-nat.apache.org (HELO spamd4-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 01 Sep 2017 13:56:07 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd4-us-west.apache.org (ASF Mail Server at spamd4-us-west.apache.org) with ESMTP id DF831C3884 for ; Fri, 1 Sep 2017 13:56:06 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd4-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: -99.202 X-Spam-Level: X-Spam-Status: No, score=-99.202 tagged_above=-999 required=6.31 tests=[KAM_ASCII_DIVIDERS=0.8, RP_MATCHES_RCVD=-0.001, SPF_PASS=-0.001, USER_IN_WHITELIST=-100] autolearn=disabled Received: from mx1-lw-us.apache.org ([10.40.0.8]) by localhost (spamd4-us-west.apache.org [10.40.0.11]) (amavisd-new, port 10024) with ESMTP id x8imIYVmNSjR for ; Fri, 1 Sep 2017 13:56:02 +0000 (UTC) Received: from mailrelay1-us-west.apache.org (mailrelay1-us-west.apache.org [209.188.14.139]) by mx1-lw-us.apache.org (ASF Mail Server at mx1-lw-us.apache.org) with ESMTP id 48CE060DA9 for ; Fri, 1 Sep 2017 13:56:02 +0000 (UTC) Received: from jira-lw-us.apache.org (unknown [207.244.88.139]) by mailrelay1-us-west.apache.org (ASF Mail Server at mailrelay1-us-west.apache.org) with ESMTP id 7C2ECE0E2B for ; Fri, 1 Sep 2017 13:56:01 +0000 (UTC) Received: from jira-lw-us.apache.org (localhost [127.0.0.1]) by jira-lw-us.apache.org (ASF Mail Server at jira-lw-us.apache.org) with ESMTP id C32A224167 for ; Fri, 1 Sep 2017 13:56:00 +0000 (UTC) Date: Fri, 1 Sep 2017 13:56:00 +0000 (UTC) From: "Steve Loughran (JIRA)" To: common-issues@hadoop.apache.org Message-ID: In-Reply-To: References: Subject: [jira] [Updated] (HADOOP-13936) S3Guard: DynamoDB can go out of sync with S3AFileSystem::delete operation MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394 archived-at: Fri, 01 Sep 2017 13:56:10 -0000 [ https://issues.apache.org/jira/browse/HADOOP-13936?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran updated HADOOP-13936: ------------------------------------ Affects Version/s: 3.0.0-beta1 > S3Guard: DynamoDB can go out of sync with S3AFileSystem::delete operation > ------------------------------------------------------------------------- > > Key: HADOOP-13936 > URL: https://issues.apache.org/jira/browse/HADOOP-13936 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 > Affects Versions: 3.0.0-beta1 > Reporter: Rajesh Balamohan > Priority: Minor > > As a part of {{S3AFileSystem.delete}} operation {{innerDelete}} is invoked, which deletes keys from S3 in batches (default is 1000). But DynamoDB is updated only at the end of this operation. This can cause issues when deleting large number of keys. > E.g, it is possible to get exception after deleting 1000 keys and in such cases dynamoDB would not be updated. This can cause DynamoDB to go out of sync. -- This message was sent by Atlassian JIRA (v6.4.14#64029) --------------------------------------------------------------------- To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org For additional commands, e-mail: common-issues-help@hadoop.apache.org