From common-issues-return-147018-archive-asf-public=cust-asf.ponee.io@hadoop.apache.org Fri Jan 19 18:34:06 2018 Return-Path: X-Original-To: archive-asf-public@eu.ponee.io Delivered-To: archive-asf-public@eu.ponee.io Received: from cust-asf.ponee.io (cust-asf.ponee.io [163.172.22.183]) by mx-eu-01.ponee.io (Postfix) with ESMTP id 78781180607 for ; Fri, 19 Jan 2018 18:34:06 +0100 (CET) Received: by cust-asf.ponee.io (Postfix) id 68732160C49; Fri, 19 Jan 2018 17:34:06 +0000 (UTC) Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by cust-asf.ponee.io (Postfix) with SMTP id ABDDE160C36 for ; Fri, 19 Jan 2018 18:34:05 +0100 (CET) Received: (qmail 38196 invoked by uid 500); 19 Jan 2018 17:34:04 -0000 Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Delivered-To: mailing list common-issues@hadoop.apache.org Received: (qmail 38176 invoked by uid 99); 19 Jan 2018 17:34:04 -0000 Received: from pnap-us-west-generic-nat.apache.org (HELO spamd1-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 19 Jan 2018 17:34:04 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd1-us-west.apache.org (ASF Mail Server at spamd1-us-west.apache.org) with ESMTP id 39840C143F for ; Fri, 19 Jan 2018 17:34:04 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd1-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: -99.911 X-Spam-Level: X-Spam-Status: No, score=-99.911 tagged_above=-999 required=6.31 tests=[KAM_ASCII_DIVIDERS=0.8, RCVD_IN_DNSWL_LOW=-0.7, SPF_PASS=-0.001, T_RP_MATCHES_RCVD=-0.01, USER_IN_WHITELIST=-100] autolearn=disabled Received: from mx1-lw-us.apache.org ([10.40.0.8]) by localhost (spamd1-us-west.apache.org [10.40.0.7]) (amavisd-new, port 10024) with ESMTP id 6miNdLL9qnLH for ; Fri, 19 Jan 2018 17:34:03 +0000 (UTC) Received: from mailrelay1-us-west.apache.org (mailrelay1-us-west.apache.org [209.188.14.139]) by mx1-lw-us.apache.org (ASF Mail Server at mx1-lw-us.apache.org) with ESMTP id 1E4AA5F3FE for ; Fri, 19 Jan 2018 17:34:03 +0000 (UTC) Received: from jira-lw-us.apache.org (unknown [207.244.88.139]) by mailrelay1-us-west.apache.org (ASF Mail Server at mailrelay1-us-west.apache.org) with ESMTP id 3F125E0E04 for ; Fri, 19 Jan 2018 17:34:02 +0000 (UTC) Received: from jira-lw-us.apache.org (localhost [127.0.0.1]) by jira-lw-us.apache.org (ASF Mail Server at jira-lw-us.apache.org) with ESMTP id E31D121208 for ; Fri, 19 Jan 2018 17:34:01 +0000 (UTC) Date: Fri, 19 Jan 2018 17:34:01 +0000 (UTC) From: "Steve Loughran (JIRA)" To: common-issues@hadoop.apache.org Message-ID: In-Reply-To: References: Subject: [jira] [Commented] (HADOOP-15183) S3Guard store becomes inconsistent after partial failure of rename MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394 [ https://issues.apache.org/jira/browse/HADOOP-15183?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16332584#comment-16332584 ] Steve Loughran commented on HADOOP-15183: ----------------------------------------- Attached: debug level log of operations leading to failure. Search for "Renaming readonly files" to get the directory rename sequence which is failing. One of the files being renamed "readonlyChild" was unsuccessfully renamed earlier and then destDir/readonlyChild deleted > S3Guard store becomes inconsistent after partial failure of rename > ------------------------------------------------------------------ > > Key: HADOOP-15183 > URL: https://issues.apache.org/jira/browse/HADOOP-15183 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 > Affects Versions: 3.0.0 > Reporter: Steve Loughran > Priority: Major > Attachments: org.apache.hadoop.fs.s3a.auth.ITestAssumeRole-output.txt > > > If an S3A rename() operation fails partway through, such as when the user doesn't have permissions to delete the source files after copying to the destination, then the s3guard view of the world ends up inconsistent. In particular the sequence > (assuming src/file* is a list of files file1...file10 and read only to caller) > > # create file dest/file1 > # delete file dest/file1 > # rename src/file* dest/ > You will not see file1 in the listing, because it will have a tombstone marker and the update at the end of the rename() didn't take place: the old data is still there. -- This message was sent by Atlassian JIRA (v7.6.3#76005) --------------------------------------------------------------------- To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org For additional commands, e-mail: common-issues-help@hadoop.apache.org