hadoop-common-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "ASF GitHub Bot (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HADOOP-14971) Merge S3A committers into trunk
Date Fri, 27 Oct 2017 12:14:00 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-14971?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16222254#comment-16222254

ASF GitHub Bot commented on HADOOP-14971:

Github user steveloughran commented on a diff in the pull request:

    --- Diff: hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3AFileSystem.java
    @@ -1130,17 +1359,23 @@ private void blockRootDelete(String key) throws InvalidRequestException
        * Perform a bulk object delete operation.
        * Increments the {@code OBJECT_DELETE_REQUESTS} and write
        * operation statistics.
    +   * Retry policy: retry untranslated; delete considered idempotent.
        * @param deleteRequest keys to delete on the s3-backend
        * @throws MultiObjectDeleteException one or more of the keys could not
        * be deleted.
        * @throws AmazonClientException amazon-layer failure.
    +  @Retries.RetryRaw
       private void deleteObjects(DeleteObjectsRequest deleteRequest)
    -      throws MultiObjectDeleteException, AmazonClientException {
    +      throws MultiObjectDeleteException, AmazonClientException, IOException {
    -    incrementStatistic(OBJECT_DELETE_REQUESTS, 1);
         try {
    -      s3.deleteObjects(deleteRequest);
    +      invoker.retryUntranslated("delete",
    --- End diff --
    Fundamentally, the core contract of a filesystem directory delete: it changes from the
state "is" to the state "isnt", in an O(1) atomic op, is not satisified, we just pretend it
    There's already ambiguity in our pretence. If an object disappears during a delete, that
is not detected. If a new object is added if it is in a large dir and a later list() call
finds it, it is deleted. Otherwise, it remains. Which means that we don't even detect updates
to a path, not unless after a delete() you do another list to see if there has been a change
(or do some leasing in dynamo, but let's not go there).
    AFAIK, delete() succeeds if any child is taken away; the outcome on an add is undefined.
rename is a bit more ambiguous w.r.t either.

> Merge S3A committers into trunk
> -------------------------------
>                 Key: HADOOP-14971
>                 URL: https://issues.apache.org/jira/browse/HADOOP-14971
>             Project: Hadoop Common
>          Issue Type: Sub-task
>          Components: fs/s3
>    Affects Versions: 3.0.0
>            Reporter: Steve Loughran
>            Assignee: Steve Loughran
> Merge the HADOOP-13786 committer into trunk. This branch is being set up as a github
PR for review there & to keep it out the mailboxes of the watchers on the main JIRA

This message was sent by Atlassian JIRA

To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org

View raw message