hadoop-common-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "ASF GitHub Bot (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HADOOP-15409) S3AFileSystem.verifyBucketExists to move to s3.doesBucketExistV2
Date Sun, 09 Dec 2018 16:55:00 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-15409?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16714020#comment-16714020
] 

ASF GitHub Bot commented on HADOOP-15409:
-----------------------------------------

Github user steveloughran commented on the issue:

    https://github.com/apache/hadoop/pull/367
  
    Oddly enough, one of the stack traces in troubleshooting_s3a shows doesBucketExist doing
the ACK check, 
    ```
    zonaws.http.AmazonHttpClient$RequestExecutor.access$500(AmazonHttpClient.java:667)
      at com.amazonaws.http.AmazonHttpClient$RequestExecutionBuilderImpl.execute(AmazonHttpClient.java:649)
      at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:513)
      at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:4229)
      at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:4176)
      at com.amazonaws.services.s3.AmazonS3Client.getAcl(AmazonS3Client.java:3381)
      at com.amazonaws.services.s3.AmazonS3Client.getBucketAcl(AmazonS3Client.java:1160)
      at com.amazonaws.services.s3.AmazonS3Client.getBucketAcl(AmazonS3Client.java:1150)
      at com.amazonaws.services.s3.AmazonS3Client.doesBucketExist(AmazonS3Client.java:1266)
      at org.apache.hadoop.fs.s3a.S3AFileSystem.lambda$verifyBucketExists$1(S3AFileSystem.java:367)
    ```
    
    The sdk may have switched a long time ago


> S3AFileSystem.verifyBucketExists to move to s3.doesBucketExistV2
> ----------------------------------------------------------------
>
>                 Key: HADOOP-15409
>                 URL: https://issues.apache.org/jira/browse/HADOOP-15409
>             Project: Hadoop Common
>          Issue Type: Sub-task
>          Components: fs/s3
>    Affects Versions: 3.1.0
>            Reporter: Steve Loughran
>            Priority: Blocker
>
> in S3AFileSystem.initialize(), we check for the bucket existing with verifyBucketExists(),
which calls s3.doesBucketExist(). But that doesn't check for auth issues. 
> s3. doesBucketExistV2() does at least validate credentials, and should be switched to.
This will help things fail faster 
> See SPARK-24000



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org


Mime
View raw message