hadoop-common-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Steve Loughran (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HADOOP-14620) S3A authentication failure for regions other than us-east-1
Date Wed, 05 Jul 2017 09:38:00 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-14620?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16074495#comment-16074495

Steve Loughran commented on HADOOP-14620:

the sole difference is that in the second case the per-bucket option is being copied in on
top of  the default fs.s3a.endpoint option...we've added that precisely so you can define
things like different endpoints for different buckets. The default endpoint value in {{fs.s3a.endpoint}}
is the one which gets used when there isn't a per bucket override going on,

If you've got the time, stepping through what's going on in S3A would be useful. I suspect
maybe there's a default value somewhere in your site configs, or indeed, the core-default
one, which is not letting the one you've set on the classpath through. Of course, you know
have an immediate fix to your problem...

> S3A authentication failure for regions other than us-east-1
> -----------------------------------------------------------
>                 Key: HADOOP-14620
>                 URL: https://issues.apache.org/jira/browse/HADOOP-14620
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: fs/s3
>    Affects Versions: 2.8.0, 2.7.3
>            Reporter: Ilya Fourmanov
>         Attachments: s3-403.txt
> hadoop fs s3a:// operations fail authentication for s3 buckets hosted in regions other
than default us-east-1
> Steps to reproduce:
> # create s3 bucket in eu-west-1
> # Using IAM instance profile or fs.s3a.access.key/fs.s3a.secret.key run following command:
> {code}
> hadoop --loglevel DEBUG  -D fs.s3a.endpoint=s3.eu-west-1.amazonaws.com  -ls  s3a://your-eu-west-1-hosted-bucket/

> {code}
> Expected behaviour:
> You will see listing of the bucket
> Actual behaviour:
> You will get 403 Authentication Denied response for AWS S3.
> Reason is mismatch in string to sign as defined in http://docs.aws.amazon.com/AmazonS3/latest/dev/RESTAuthentication.html
provided by hadoop and expected by AWS. 
> If you use https://aws.amazon.com/code/199 to analyse StringToSignBytes returned by AWS,
you will see that AWS expects CanonicalizedResource to be in form  /your-eu-west-1-hosted-bucket{color:red}.s3.eu-west-1.amazonaws.com{color}/.
> Hadoop provides it as /your-eu-west-1-hosted-bucket/
> Note that AWS documentation doesn't explicitly state that endpoint or full dns address
should be appended to CanonicalizedResource however practice shows it is actually required.
> I've also submitted this to AWS for them to correct behaviour or documentation.

This message was sent by Atlassian JIRA

To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org

View raw message