hadoop-common-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Steve Loughran (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HADOOP-15183) S3Guard store becomes inconsistent after partial failure of rename
Date Mon, 05 Mar 2018 12:49:00 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-15183?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16386026#comment-16386026

Steve Loughran commented on HADOOP-15183:

committed; thanks for fixing this up!

> S3Guard store becomes inconsistent after partial failure of rename
> ------------------------------------------------------------------
>                 Key: HADOOP-15183
>                 URL: https://issues.apache.org/jira/browse/HADOOP-15183
>             Project: Hadoop Common
>          Issue Type: Sub-task
>          Components: fs/s3
>    Affects Versions: 3.0.0
>            Reporter: Steve Loughran
>            Assignee: Steve Loughran
>            Priority: Blocker
>         Attachments: HADOOP-15183-001.patch, HADOOP-15183-002.patch, org.apache.hadoop.fs.s3a.auth.ITestAssumeRole-output.txt
> If an S3A rename() operation fails partway through, such as when the user doesn't have
permissions to delete the source files after copying to the destination, then the s3guard
view of the world ends up inconsistent. In particular the sequence
>  (assuming src/file* is a list of files file1...file10 and read only to caller)
> # create file rename src/file1 dest/ ; expect AccessDeniedException in the delete, dest/file1
will exist
> # delete file dest/file1
> # rename src/file* dest/  ; expect failure
> # list dest; you will not see dest/file1
> You will not see file1 in the listing, presumably because it will have a tombstone marker
and the update at the end of the rename() didn't take place: the old data is still there.

This message was sent by Atlassian JIRA

To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org

View raw message