hadoop-common-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Steve Loughran (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HADOOP-15409) S3AFileSystem.verifyBucketExists to move to s3.doesBucketExistV2
Date Fri, 27 Jul 2018 21:40:00 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-15409?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16560366#comment-16560366

Steve Loughran commented on HADOOP-15409:

Looking into the AWS SDK here, the v2 call does getBucketAcl(); if the user doesn't have the
perms to query the permissions it is still treated as proof of existence. The big diff here
is that permissions are checked early, which is good.

One thing I worry about: do all S3 re-implementations implement this call? Does anyone know

> S3AFileSystem.verifyBucketExists to move to s3.doesBucketExistV2
> ----------------------------------------------------------------
>                 Key: HADOOP-15409
>                 URL: https://issues.apache.org/jira/browse/HADOOP-15409
>             Project: Hadoop Common
>          Issue Type: Sub-task
>          Components: fs/s3
>    Affects Versions: 3.1.0
>            Reporter: Steve Loughran
>            Priority: Blocker
> in S3AFileSystem.initialize(), we check for the bucket existing with verifyBucketExists(),
which calls s3.doesBucketExist(). But that doesn't check for auth issues. 
> s3. doesBucketExistV2() does at least validate credentials, and should be switched to.
This will help things fail faster 
> See SPARK-24000

This message was sent by Atlassian JIRA

To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org

View raw message