hadoop-common-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Aaron Fabbri (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HADOOP-14391) s3a: auto-detect region for bucket and use right endpoint
Date Fri, 05 May 2017 19:39:04 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-14391?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15998852#comment-15998852

Aaron Fabbri commented on HADOOP-14391:

There appears to be [an API|http://docs.aws.amazon.com/AWSJavaSDK/latest/javadoc/com/amazonaws/services/s3/AmazonS3.html#getBucketLocation-com.amazonaws.services.s3.model.GetBucketLocationRequest-]
 to query bucket location.

I suppose we could:

- Introduce a new config flag, say, {{fs.s3a.auto.set.endpoint}} (or we could use a special
value "auto" for {{fs.s3a.endpoint}}.)
- When enabled, at init time, use {{AmazonS3#getBucketLocation(..)}}, {{Region#fromValue()}},
then build the endpoint name s3-${REGION}.amazonaws.com and set it.

> s3a: auto-detect region for bucket and use right endpoint
> ---------------------------------------------------------
>                 Key: HADOOP-14391
>                 URL: https://issues.apache.org/jira/browse/HADOOP-14391
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: s3
>    Affects Versions: 3.0.0-alpha2
>            Reporter: Aaron Fabbri
> Specifying the S3A endpoint ({{fs.s3a.endpoint}}) is
> - *required* for regions which only support v4 authentication
> - A good practice for all regions.
> The user experience of having to configure endpoints is not great.  Often it is neglected
and leads to additional cost, reduced performance, or failures for v4 auth regions.
> I want to explore an option which, when enabled, auto-detects the region for an s3 bucket
and uses the proper endpoint.  Not sure if this is possible or anyone has looked into it yet.

This message was sent by Atlassian JIRA

To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org

View raw message