hadoop-common-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "ASF GitHub Bot (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HADOOP-14971) Merge S3A committers into trunk
Date Thu, 26 Oct 2017 14:10:00 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-14971?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16220474#comment-16220474

ASF GitHub Bot commented on HADOOP-14971:

Github user steveloughran commented on a diff in the pull request:

    --- Diff: hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/InconsistentAmazonS3Client.java
    @@ -426,8 +490,10 @@ private void registerDeleteObject(String key, String bucket) {
         if (shouldDelay(key)) {
           // Record summary so we can add it back for some time post-deletion
           S3ObjectSummary summary = null;
    -      ObjectListing list = listObjects(bucket, key);
    -      for (S3ObjectSummary result : list.getObjectSummaries()) {
    +      ListObjectsV2Request request =
    +          new ListObjectsV2Request().withBucketName(bucket)
    +              .withPrefix(key);
    --- End diff --
    No real reason. I do recall how an aggressive throttle rate had real problems with listing
because a single op could fail too many times for the retry handlers to cope. But I don't
see that this has fixed it because its not calling the innerList() op which is needed to bypass
that inconsistency. 
    I'll revert to v2 and innerListV1, so that there's no failures in this loop, and, assuming
all stores support v1 alongside v2, work everywher

> Merge S3A committers into trunk
> -------------------------------
>                 Key: HADOOP-14971
>                 URL: https://issues.apache.org/jira/browse/HADOOP-14971
>             Project: Hadoop Common
>          Issue Type: Sub-task
>          Components: fs/s3
>    Affects Versions: 3.0.0
>            Reporter: Steve Loughran
>            Assignee: Steve Loughran
> Merge the HADOOP-13786 committer into trunk. This branch is being set up as a github
PR for review there & to keep it out the mailboxes of the watchers on the main JIRA

This message was sent by Atlassian JIRA

To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org

View raw message