hadoop-common-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "ASF GitHub Bot (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HADOOP-15409) S3AFileSystem.verifyBucketExists to move to s3.doesBucketExistV2
Date Sun, 09 Dec 2018 17:17:00 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-15409?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16714026#comment-16714026
] 

ASF GitHub Bot commented on HADOOP-15409:
-----------------------------------------

Github user steveloughran commented on the issue:

    https://github.com/apache/hadoop/pull/367
  
    Next patch for this will give you a choice of v1 or v2, so that third-party endpoints
will work. Versions other than 1 or 2 get told off and then continue
    ```
     bin/hadoop fs -D fs.s3a.bucket.exists.version=3 -ls s3a://fdsd/
    2018-12-09 17:11:58,015 WARN util.NativeCodeLoader: Unable to load native-hadoop library
for your platform... using builtin-java classes where applicable
    2018-12-09 17:11:58,537 INFO impl.MetricsConfig: loaded properties from hadoop-metrics2.properties
    2018-12-09 17:11:58,598 INFO impl.MetricsSystemImpl: Scheduled Metric snapshot period
at 10 second(s).
    2018-12-09 17:11:58,598 INFO impl.MetricsSystemImpl: s3a-file-system metrics system started
    2018-12-09 17:11:59,177 WARN s3a.S3AFileSystem: Unknown fs.s3a.bucket.exists.version version:
3
    2018-12-09 17:12:02,729 INFO Configuration.deprecation: fs.s3a.server-side-encryption-key
is deprecated. Instead, use fs.s3a.server-side-encryption.key
    ls: s3a://fdsd/: getFileStatus on s3a://fdsd/: com.amazonaws.services.s3.model.AmazonS3Exception:
All access to this object has been disabled (Service: Amazon S3; Status Code: 403; Error Code:
AllAccessDisabled; Request ID: 7D34514B3633D1EC; S3 Extended Request ID: AWimKqhxDqr10Q22QVnh71xN/ctC9UK+x7/FubEbbMIs8sjpXbTrCttzy9r1BUvUKeFM0E9Jlb8=),
S3 Extended Request ID: AWimKqhxDqr10Q22QVnh71xN/ctC9UK+x7/FubEbbMIs8sjpXbTrCttzy9r1BUvUKeFM0E9Jlb8=:AllAccessDisabled
    2018-12-09 17:12:03,163 INFO impl.MetricsSystemImpl: Stopping s3a-file-system metrics
system...
    2018-12-09 17:12:03,163 INFO impl.MetricsSystemImpl: s3a-file-system metrics system stopped.
    2018-12-09 17:12:03,163 INFO impl.MetricsSystemImpl: s3a-file-system metrics system shutdown
complete.
    ```


> S3AFileSystem.verifyBucketExists to move to s3.doesBucketExistV2
> ----------------------------------------------------------------
>
>                 Key: HADOOP-15409
>                 URL: https://issues.apache.org/jira/browse/HADOOP-15409
>             Project: Hadoop Common
>          Issue Type: Sub-task
>          Components: fs/s3
>    Affects Versions: 3.1.0
>            Reporter: Steve Loughran
>            Priority: Blocker
>
> in S3AFileSystem.initialize(), we check for the bucket existing with verifyBucketExists(),
which calls s3.doesBucketExist(). But that doesn't check for auth issues. 
> s3. doesBucketExistV2() does at least validate credentials, and should be switched to.
This will help things fail faster 
> See SPARK-24000



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org


Mime
View raw message